Spawn of Ten Thousand Websites
A web site for assisting in the creation of web sites.
By William Perilli
This document is private and confidential.
The information in this document is trade secrets of Amillia Publishing Company. Disclosure is limited.
Copyright 2004, Amillia Publishing Company.
Duplication and dessemination is prohibited.
Have arrainged for broadband internet connection and new phone line into my house, as well as cable TV. Installation will be on Friday. The logistics of setting up the system are still being worked out.
The addition of broadband create the need for a firewall/router at my house so that all of my machines will be able to use the internet.
I have investigated the idea of getting another Linux machine and running things like that, but a much lower cost item would something like a Linksys BEFSX41. Listed price in Computer Shopper (October, 2004) is $69.
I am continuing my investigation of this and will get something hopefully today or tomorrow.
I talked with Peter Mark about my continuing work. He gave me a lot of different ideas and answered a lot of my questions.
I am working on PHP modules to do the bulk of what I need to do. I have made a lot of progress on getting reasonable things going on.
I took a bunch of notes when I talked Peter. Some of what I asked him was about his set up and how he does things.
He uses Apache and Perl.
He uses a Mod perl type thing that keeps his modules in memory (for bandwidth)
He says that using tables is preverable to cascading style sheets for the html output to the user (as generated by the Perl scripts in his case and PHP scripts in my case).
He detailed the ability to charge credit cards. He said that the way he does it requires (he believes) incorporation. So he doesn't recommend that I do things that way. There are services that will take a cut of the charges and thus make it very easy for me to get credit card service. This is an open issue and an on-going concern.
I asked him how he saves data for later serveing back to the user. He does it through a perl module that writes perl scripts that then spit the data back in a format that he can use! This sounds like a very convenient way to do things. It does away with the need for XML or databases.
Peter will, at client requests, provide methodology for them to get xml back if they want it (for a fee I suppose).
The name of the perl module that he uses is for this data interface:
datadumper
I amy also be able to use this or find a PHP equivalent.
Hosting service that he uses:
Rackspace
He recommends a company called Hostrocket which someone he trusts recommended to him. I need to look at both and choose.
I asked him the question about OS and if it matters. He said it does. I agreed with him. He likes freeBSD but as I already use Linux I might just decide to go with that.
He has optional https. I imagine that the clients might need to pay more for that. As far as I am concerned I do not need https unless I am doing a transaction. Peter mentioned that credit card number acquisition is best doen through a service like paypal. I knew this.
Verisign is what he uses for $100/month with more charge if more than 2000 transactions. This would be overkill for me at first.
I asked Peter about the difference between his post and get algorithms. Basically he proxies either into a single variable and that way he is able to have one set of modules that do all the work of spawning the pages out. Typically the first page is a different method than the subsequent. And so for AOL users who acess his page they must use cookies.
Peter provides a cookie and a POST/GET interface.
The GET functions can be bandwidth limited. So in that case if there is a lot of data it must be passed with a POST.
As I want to provide editing and database functionality to provide a way for my website holders to modifiy their pages and products, I will need to work on this further.
More than a 1000 bytes needs a POST.
I asked about style sheets. I told him how I had been investigating their use verses the use of tables. Tables work in all browsers. They provide the most cross-browser capabilities. Old browsers do not do css style sheets well. But he does use both with the table as his prefered way.
I have had so much success with tables in word processors and in application programming that it seems obvious to me that this is what I will have as the bedrock of my page layout architecture.
He tests his pages with Internet explorer and with Firefox.
Style sheet layout often has 'edge conditions' Peter told me.
I can get a domain name in a lot of differnet ways.
There are the following companies:
Verio.
When I get the server verses when I get the domain name doesn't matter, but it might be easiest to get them all at the same time.
He uses either ssh or secureFTP to connect to his servers from his development environment.
He says that once things are up and running that it is very easy to maintain things. He uses Fast CGI and scripts written in Perl.
For plugins he uses C/C++ .
I have gotten information on html from the html consortium. I have created a more advanced version of the web photo viewer that I wrote in php by modifying the stuff that I had before.
I am learning about style sheets and tables as well as how the newer protocols handle some of the issues that I have. Obviously there is what is suggested, what is possible, and what people are actually using. It is important to learn about the different ways that people are able to do what they need to do. Essentially it is best for me to learn how to do things in a way that will be efficient and easy for me and not necessarily something that embraces all of the latest protocols and technical fads.
Currently I have a 'helpers.php' file that is a lot of code.
I have to do this more logically.
The question about using xml or sql comes up. There are really good reasons for using sql. Basically the ability to have two users at once querrying the data in a database really rocks. However, in the case of 'spawn of ten thousand websites', which is what my site wants to be, there are needs for public read only data, and also private read/write data.
The need for both of these types of data is clear for a bunch of reasons. First we have to consider the creator of a site. The creator, or author, of a site needs to be able to catelog the elements of the site. There is the need to have entities available for display. The configuration and presentation of these entities is the sole responsibility of the creator.
There will be multiple creators and they will interact. Creators need to know that their sites will not be compromised by other creators. Thus they need to have their own databases for themselves, and then common ones that they all use.
There are a bunch of different ways to model this.
And then on top of all of that there are underlying construction and style elements that need to be available to all users. These elements should be shared, though it may suite some users to have some of their input to this as part of what they do and only those who they let use their styles.
For example a designer might have some very cool styles that are developed with great care. This designer should benefit if others extensively use these created styles. And then there could be a cache associated with the use of such a designers styles. So in this sense we could have the ability to allow/disallow the use of certain types of styles based upon a license, a subscription, or possibly membership in the group.
The catalog designer styles is available to the creators or to the users that they let select the ones that they have available.
A creator can use a style when that creator has permission. The implementation of a style (CSS or attribute based definitions) will be handled in the background by the scripts and the architecture of the 'spawn of ten thousand websites' site.
Here is a picture of what things look like in the very alpha state. Date of this shot is Oct 18, 2004
Each of the thumbnails is a link that will thus show that picture as the main picuture. The user can select the gallery from the form input, but the form input is not designed so that the user can set it up easily if the page is reached from a get that doesn't have any information in it. I don't use cookies or sessions. There is no logging on.
I would like to add the following features:
The ability to comment the picture
The ability to creat new groups for the Creators
The ability to add pictures.
Use of the pictures in other presentations of data, ie as a way to annotate other lists that the user also creates.
Here are a couple more screen shots:
Discussion of Materials
Part of doing the webpage design is understanding what methods are available. And also I don't want to have to reinvent everything all of the time, either. If there are tools that I can already use, then I would like to use these or at least understand why I shouldn't use them. But as there are a lot of different things available there is a point where it seems as if the learning about what is available becomes overwhelming. After all I cannot use all of the tools, and I do need to finish something.
I am creating a dynamic content manager and website spawner. This system will produce dynamic content. I am not yet using scripting within the page itself, but server side scripting instead with Apache (also called httpd). It has seemed useful to me that I should have example pages that show me various idioms of html. A lot of the newer stylistic elements of html are not adopted by all writers. As I explained elsewhere tables can be stylized with cascading style sheets, and layout can be accomplished that way too. However html is not designed to do exact layout. And css is something that a lot of folks say one does not need to use. I feel that it is prudent to investigate it. However, the output of a dynamic content generator can often be hard to grasp. I have reviewed the source code of a bunch of pages that I like and noticed some of the styles that they used. And I found the html 4.0 spec and started to read the examples in that. And I took some of the information out of that and used to to see what things look like.
From that I am taking stylistic leads. Clearly the use of tables and CSS is prefered, and people have not given up on tables at all. Also the use of frames is not discouraged in the materials I found, but I happen to know that people don't use them. Thus I have to question the direction that the documents that I have read point me. And instead I need to look further and learn more about what people really do that works.
There is a paralysis that a pedantic person can have that is the result of having a large amount of information and a fear of making the wrong decision. And then necessity is the mother of invention. And so we learn a lot of things and then when we have to, suddenly there we are doing it the easiest way possible when we finally have a crisis. In that sense it behooves us to keep learning. But on the other hand one who does is more likely to succeed. Not all attempts at doing something pan out. Aim, and fire. One must not just aim or just fire.
And so the prudent designer will spend a lot of time thinking, designing, and deciding. And the designer will try to 'do' as little as possible in order to get what needs to be done. That does not mean that the creator is not busy learning new things and 'doing' as much as possible. But the designer aims high and for each little piece of doing a lot tries to do the least that quality allows.
Obviously the designer can lavish on some small part. But then this takes away from all else.
In the extravagant world of mainline companies there are all of these people who do a very small thing. And thus they get 'group think' output. The designer is often working alone or in a small group. There is not the urgency or luxury of being able to specialize. Yes the designer is a 'specialist'. However a generalist in all of the various tools that are available.
I will keep in billbin an area that has example static html files. I have started to do this. There will be static examples as well within the website that will be available. I will have links to these and encourage the users to review them as they need to.
I will create a style guide and perhaps also link to other guides. I will explain in my guide the current philosophy of Amillia Publishing spawn of ten thousand websites page designer. I will put some of the material that I develop there, as well as create a guide to using the website in general to create a new website for the client.
All of this leads to a very real and compeling need for me to make a map of what the designer will allow. I need to better lay out my architecture.
10_20_2004 in the A.M.
I worked for a long time until I came up with some reasonable (but not perfect) scripts that allow me to easily convert the files to smaller sizes.
I need to still implaint the copyrights into these files so that they are marked as mine. I wrote three file, the last one runs the second type which itteratively runs the first.
The last one is written in lieu of writing a more complex method. What I have produces the thumbnails that I need in a format that makes sense and that will allow me to easily catelog the pictues.
I have encoded in the name of the picture what the directory and filename is in which I store the large picture. Here are the scripts:
#!/bin/bash
#filename: jptopnm
#this script is designed to be run in the custom environment of Bluesky, the
#APC server. Things will evolve as needed. This script solves a lot of
#design concerns with the spawn of ten thousand websites website builder.
#copyright Amillia Publishing Company 2004
#This file consitutes a trade secret of Amillia Publishing Company.
# Use or disclousure of this file is prohibited.
#this script takes two input parameters: $1 is the name of the file
#$2 is the name of the directory.
#this script should be run from within that directory
# the script that calls this cd's to the correct directory
# and then call this for every *.jpg found.
# if the file is the correct nameing format then it
# will be converted to smaller versions in a png format.
# the file names chosen for the smaller versions are logical
# rotating and implainting copyrights and data will be added later.
#mkdir /home/bperil/billbin/%1
#cd /home/bperil/billbin/
echo script for scaling and converting.
#echo $2$1
#echo next step
FINAME=`echo $1 |sed "s/.jpg//g" | sed "s/$/.png/g"`
echo "FINAME is $FINAME"
#echo last step
echo 1
FINAME=`echo "$1" | sed "s/.jpg//g" | sed "s/$/.png/g" | sed "s/pict/$2_&/"`
#echo "FINAME is $FINAME"
#echo 2
FINAME=`echo "$FINAME" | sed "s/.\///"`
#echo "FINAME is $FINAME"
#echo 3
#FINAME=`echo $FINAME | sed "s/.jpg//g" | sed "s/$/.png/g" | sed "s/pict/$2_&/"`
#echo 3.4
#echo "FINAME is $FINAME"
#echo 3.5
#echo 4
# we do the conversion in the first step using the filename that we like
# and then we rename the file so that is named after the directory where
# it lives. This is the second command line arguement and results in
# a date being the first part of the name if adhering to the APC format.
jpegtopnm $1 |pnmscale 0.2 |pnmtopng > $FINAME
rename .png _020.png $FINAME
jpegtopnm $1 |pnmscale 0.1 |pnmtopng > $FINAME
rename .png _010.png $FINAME
jpegtopnm $1 |pnmscale 0.05 |pnmtopng > $FINAME
rename .png _005.png $FINAME
#end of bash script jptopnm
#!/bin/bash
# filename: findjpgtopnp
# This script works in concert with the jptopnm script
# This script takes a directory name (in APC date format)
# and then will convert all *.jpg files in that directory to be thumbnails
# of various sizes.
# These thumbnails will then be copied to a destination directory
# in the billbin area for later copying to the httpd bin area for serving on the
# internet.
#this script is designed to be run in the custom environment of Bluesky, the
#APC server. Things will evolve as needed. This script solves a lot of
#design concerns with the spawn of ten thousand websites website builder.
#copyright Amillia Publishing Company 2004
#This file consitutes a trade secret of Amillia Publishing Company.
# Use or disclousure of this file is prohibited.
mkdir /home/bperil/billbin/tiny_pics/$1
cd /home/bperil/Desktop/CamPics/$1
find -name "*.jpg" -print -exec ~/billbin/phpscripts/jptopnm "{}" "$1" \;
mv *.png /home/bperil/billbin/tiny_pics/$1
#rm /home/bperil/Desktop/CamPics/$1/*.png
#end of bash script findjpgtopnp
The final script which is one to run (hopefully) just once as it will run for a long time ( a few hours on my machine)
But after I do this I will have reasonable sized thumbnails that I can use for my webpages.
#!/bin/bash
#/home/bperil/billbin/phpscripts/findjpgtopnp 10_18_2004
/home/bperil/billbin/phpscripts/findjpgtopnp 09_19_2004
/home/bperil/billbin/phpscripts/findjpgtopnp 09_21_2004
/home/bperil/billbin/phpscripts/findjpgtopnp 09_23_2004
/home/bperil/billbin/phpscripts/findjpgtopnp 09_23_B_2004
/home/bperil/billbin/phpscripts/findjpgtopnp 09_26_2004
/home/bperil/billbin/phpscripts/findjpgtopnp 09_26_B_2004
/home/bperil/billbin/phpscripts/findjpgtopnp 09_27_2004
/home/bperil/billbin/phpscripts/findjpgtopnp 09_30_2004
/home/bperil/billbin/phpscripts/findjpgtopnp 10_03_2004
/home/bperil/billbin/phpscripts/findjpgtopnp 10_04_2004
/home/bperil/billbin/phpscripts/findjpgtopnp 10_04_B_2004
/home/bperil/billbin/phpscripts/findjpgtopnp 10_04_C_2004
/home/bperil/billbin/phpscripts/findjpgtopnp 10_04_D_2004
/home/bperil/billbin/phpscripts/findjpgtopnp 10_04_E_2004
/home/bperil/billbin/phpscripts/findjpgtopnp 10_06_2004
/home/bperil/billbin/phpscripts/findjpgtopnp 10_07_2004
/home/bperil/billbin/phpscripts/findjpgtopnp 10_09_2004
/home/bperil/billbin/phpscripts/findjpgtopnp 10_11_2004
/home/bperil/billbin/phpscripts/findjpgtopnp 10_11_b_2004
/home/bperil/billbin/phpscripts/findjpgtopnp 10_13_2004
/home/bperil/billbin/phpscripts/findjpgtopnp 10_17_2004
Programmers note: Not a good idea to modify scripts while they are running! I commented the various above scripts and saved them. So then I had to modify the last one, remove a directory so that the scripts wouldn't barf, and then run the script again.
jpegtopnm: WRITING PPM FILE
[bperil@bluesky phpscripts]$ ls
convert_abunch findjpgtopnp formp.php jptopnm
[bperil@bluesky phpscripts]$ ./convert_abunch
./pict0001.jpg
script for scaling and converting.
FINAME is ./pict0001.png
1
jpegtopnm: WRITING PPM FILE
jpegtopnm: WRITING PPM FILE
jpegtopnm: WRITING PPM FILE
./pict0002.jpg
script for scaling and converting.
FINAME is ./pict0002.png
1
jp
Today I wrote an html file that would show all the images in a directory at once. Composing the page was a breeze, but the page did not work as I would have liked. There were 165 or so images that were all about 1.5 meg each. The browser took for ever to render the images to the small size that i desired. In actuallity the thing froze up for quite a while. The redering on the screen was OK, but not as good as when I used the thumbnails. And it took a long time.
This was a simple file and yet it brought the browser to a virtual crawl while it chunked through all of these huge files and made the thumbnails.
Here is a fragment from this file so you can get the idea of what I did:
<img SRC ="pict0001.jpg " width=" 450" height="300" border="" alt="1">
<img SRC ="pict0002.jpg " width=" 450" height="300" border="" alt="1">
<img SRC ="pict0003.jpg " width=" 450" height="300" border="" alt="1">
<img SRC ="pict0004.jpg " width=" 450" height="300" border="" alt="1">
<img SRC ="pict0005.jpg " width=" 450" height="300" border="" alt="1">
<img SRC ="pict0006.jpg " width=" 450" height="300" border="" alt="1">
Next things to work on:
Adding lists
Adding captions
logon
credit card information
rotating images
masking images
moving between pages
setting up for distinctive and varied styles
setting up CSS feeds (which ought to be easy)
setting up display different for each picture with other pictures around when one is featured (web mueseum effect).
John and i talked today about this project and I explained to hime htt I thought that this was going to be, in itself, a piece of art. Yes we want to sell art but the experience of using our website shouldbe akin to going to a gallery. The viewers will see pleasing graphics. The website will be enjoyable to view.
After doing all of the converting I noted that the files for specific days were not converted. The reason was that they were owned by the root as they were a product of a download from the notebook from when I was traveling. And thus I changed the user to me and then ran the script again for those directories. Now the file conversion seems to be happening for those.
Du cpmmand showed me how much disk space. This showed a useage of 4 for the empty directories.
I need to reload all of the pictures that I deleted from the computer and convert those too.
For about 4 months of pictures I have about 1 and a half gig, which is a lot less than if the pictures are full sized.
The item that is for sale must have the following properties:
price // what we want for the item
old_price // if we lower price, we will indicate this
item description //
shipping and handling costs
item number
UPC product code
condition
warranty information
item owner
applicabile tax information
link to further information for what ever else is needed for the item.
Item location
item availability
accessories links for the item (a list)
link to picture or display of item
style for the presentation of the item.
Links to other items that may also be of interest
link to return policy for item
link to reviews
link to customer support
link to third party information
It will further the design of the data f I qualify the use of this information. In data table there will be required and optional fields. I need a system in which to code this data. SQL table definitions may fit the purpose exactly.
A link is such a key element in this system that it deserves special treatment. The link is a key data element of any web or network based system. It helps us to give it special attention and special treatment.
Every link should have data concerning suggested presentation of the link in question. For example a link to a company might have the companies logo. And links should have alternative information and a link name.
The paper selector allows the website creator to select patterns for background images.
Click on an image.
Click on another part of the image.
This creates a box that allows the creation of paper. The span of what was selected is copy from the image
Frame style selection is something that website creator will be allowed to do. This will be a series of selections
Frame creation will be available at some time, however I have not yet concieved of how to do this. Obviously I know how to do it by creating code. But how do I then get from there to the ability to add content into frames. I need rules on how deep the levels should go. It would make sense to just let the user do what he wants. If the interface 'breaks' then the user will know because stuff won't come out as expected. HTML agents do a good job of fixing bad html code. If there are 'too many' levels of depth, then the user will just have to deal with that.
The following idiom does a good job of inserting content into a cell:
$c = curl_init("http://localhost/webresBillP.html");
//curl_setopt($c,CURLOPT_RETURNTRANSFER,1);
//curl_setopt($c,CURLOPT_COOKIEJAR,$cookie_jar);
//curl_setopt($c,CURLOPT_COOKIEFILE,$cookie_jar);
$subsequent_page = curl_exec($c);
curl_close($c);
print ($subsequent_page);
Copyright
2004. page