Imdb api
I have created a Ruby code that gathers information from IMDB and presents it in a nice class. The code uses why's hpricot v0.4.76.
Creating an object. The HTML is only loaded once as long as you have the object.
If using the IMDB class in a command line, the menu option will present you with the titles and years that are listed on the IMDB search site and you enter a number. If there is menu (default value is false) it will pick the first link found. If no links are found it raises an error
Now lets say you want to print the list of actors.
Although most movies do not have a plot IMDB still tries to list one.
So far its not to hard. BUT BECKER!! I want the poster!
Have no FEAR (some what)!
This will return the a img html tag, I am working on making this better.
Some other things.
IMDB has no published API. I have found no other sites providing a good API. So I made one.
http://stephenbeckeriv.com/movie/office_space
The title uses _ for spaces. You can also use %20 but I made it easy. You will have uri encode your titles.
Also if you are not getting the search result you want, try looking at the returning titles.
http://stephenbeckeriv.com/movie_titles/office_space
Please email me with comments and suggestions!
Creating an object. The HTML is only loaded once as long as you have the object.
movie=IMDB.new("monty python holy grail") or menu=true movie=IMDB.new("monty python holy grail",menu)
If using the IMDB class in a command line, the menu option will present you with the titles and years that are listed on the IMDB search site and you enter a number. If there is menu (default value is false) it will pick the first link found. If no links are found it raises an error
Now lets say you want to print the list of actors.
.actors is a hash where the key is he actor's name and the hash is the list of roles(CVS).movie.actors.each{|name,role| p name+" starred in "+ movie.title+" as "+role+"\n" }
Although most movies do not have a plot IMDB still tries to list one.
p "This movies plot might be like: "+movie.plot
So far its not to hard. BUT BECKER!! I want the poster!
Have no FEAR (some what)!
movie.poster
This will return the a img html tag, I am working on making this better.
Some other things.
movie.director movie.run_time movie.year movie.genres #an array movie.mpaa #html with rating and reason movie.to_xml#ohh why would I do this?
IMDB has no published API. I have found no other sites providing a good API. So I made one.
http://stephenbeckeriv.com/movie/office_space
The title uses _ for spaces. You can also use %20 but I made it easy. You will have uri encode your titles.
Also if you are not getting the search result you want, try looking at the returning titles.
http://stephenbeckeriv.com/movie_titles/office_space
Please email me with comments and suggestions!
Site map
Looking around google's Webmaster Tools I found a well hidden link to google's help page about site map formatting. Looking at the Sitemap protocol (just a simple xml file) I set off to create a method that would take my dynamic website and create a nice little map for it. After all if you can not find it in google, you really don't need it.
I am sure I can come up with a general method to do this, but currently it is specific to how I have created my website.
You have to start your xml docuemnt. I really like using rexml/document, you will also need zlib class
And you are done! Make this a method on your admin console and submit the link to google. A few things I want to do with this is provide a link to the xml, and create a dynamic site map with ajax using the same ideas. If someone wanted to present a map to my site they could use the xml and format it how they like. I am thinking about creating a general map that looks at the views and routes.rb to try and create the proper links.
Update:
Turns out when i added my site to the webmaster tools in google i used http://stephenbeckeriv.com/ not http://www.stephenbeckeriv.com/ which it does not like my site map because I added the www. How fun and flexible.
I am sure I can come up with a general method to do this, but currently it is specific to how I have created my website.
You have to start your xml docuemnt. I really like using rexml/document, you will also need zlib class
data=REXML::Document.new('<?xml version="1.0" encoding="UTF-8"?>') base=data.add_element("urlset") base.attributes["xmlns"]="http://www.google.com/schemas/sitemap/0.84"</pre> <br>These values are found on the Sitemap protocol website<br><br> Then I add in my static content values<br><pre> el=base.add_element("url") el.add_element("loc").text="http://www.stephenbeckeriv.com/" #Time needs to be in w3c formatting, For the life of me I can not use Time's w3cdtf el.add_element("lastmod").text=Time.now.strftime("%Y-%m-%d") el.add_element("changefreq").text="weekly" el.add_element("priority").text="0.5" #turns out they do not like svn sites #el=base.add_element("url") #el.add_element("loc").text="http://www.svn.stephenbeckeriv.com/code" #el.add_element("lastmod").text=Time.now.strftime("%Y-%m-%d") #el.add_element("changefreq").text="weekly" #el.add_element("priority").text="0.5"</pre> <BR><BR>Then I add in the context part of my site<br> <pre> @context_list = Post.find(:all,:select=>"context") arr=[] @context_list.each{|x|arr.push(x.context)} arr.uniq.each{|context| el=base.add_element("url") el.add_element("loc").text="http://www.stephenbeckeriv.com/#{context}" el.add_element("lastmod").text=Time.now.strftime("%Y-%m-%d") el.add_element("changefreq").text="weekly" el.add_element("priority").text="0.5" } </pre> <br> I know I can uniq my results with the sql, but I could not remember the code for this. <br><BR> Then I add my dynamic content<br><pre> @post=Post.find(:all) @post.each{|post| el=base.add_element("url") #if you read my first post you know why this works el.add_element("loc").text="http://www.stephenbeckeriv.com/#{URI.encode(post.title.gsub(" ","_"))}" a=Time.parse("#{post.created}")#it did not like having the date formatting on one line. el.add_element("lastmod").text=a.strftime("%Y-%m-%d") el.add_element("changefreq").text="never" el.add_element("priority").text="0.8" }</pre> <br><BR>We Now have all the web pages for my site. Now we need to save this to a file in public. There are some limits for the file. You can not have over 500,000 urls, the file must be less then 10MB. If this is a problem, you have to use the site map index files. I do not cover this. <br><br><pre> result="" data.write(result) dir=Dir.pwd #local dir.does not have /public dir<< "/public/sitemap.xml.gz" if !dir.include("/public") #server does have public dir<< "/sitemap.xml.gz" if dir.include("/public") Zlib::GzipWriter.open(Dir.pwd+"/public/sitemap.xml.gz"){|file| file.write result }
And you are done! Make this a method on your admin console and submit the link to google. A few things I want to do with this is provide a link to the xml, and create a dynamic site map with ajax using the same ideas. If someone wanted to present a map to my site they could use the xml and format it how they like. I am thinking about creating a general map that looks at the views and routes.rb to try and create the proper links.
Update:
Turns out when i added my site to the webmaster tools in google i used http://stephenbeckeriv.com/ not http://www.stephenbeckeriv.com/ which it does not like my site map because I added the www. How fun and flexible.
Prepared statements
I have set out to create prepared statement support for active record. So far it is looking good. I am looking at different ways to manage the statements. The biggest pain will be supporting the syntax of different databases. I will start out with statements in Postgres.
Rails conf 2007
It's official I will be speaking at Rails Conf 2007!
Rails conf bio
Stephen Becker IV is a Cancer with a a smooth demeanor and a voice like velour. His turn-ons are long walks on the beach, futurama and all night caffeine induced bing programming sessions. His turn-offs are stupidity and the lack of velour clothing currently in stores. Presently working for Vonage as a rails developer in New Jersey.
Rails Conf link
Rails Conf link
Prepared statements start
Using prepared statements in your applications is like giving a tug on your spice weasel!
I am not going to talk about why prepare statements are important. That information can be found at google, or from years of doing database work. This is how I am going add support for them in to active record.
The syntax is not complete yet but it will go down something like this...
I think its simple and sweet. We are also working on "auto_magic" mode for people who do not know sql. This will high jack active records processes of creating the sql, and create statements, manage them and run them instead of the normal sql statements. I just want to make sure no one can say I discriminate against people who do not know sql. Also, I have yet to test what happens if you call the a statement with the wrong model. Currently we do not have the concept of this model owns this statement. We need a better handling of errors and more testing with Oracle. But as Nick always says "Its piratically done"
I am not going to talk about why prepare statements are important. That information can be found at google, or from years of doing database work. This is how I am going add support for them in to active record.
The syntax is not complete yet but it will go down something like this...
ActiveRecord::Base.create_ps({:key=>'Pstatemnet1', :sql=>'select * from cost_to_cost where id=$1;}) </code <br> <br><br>Notice the use of very helpful names. Also, notice the ps and not prepared_statement the less I have to type the more I will use it, and it will just be an ailis anyway. So, key is going to be the name that is associated with the statement. As I currently understand it postgres allows you to set this and oracle slaps you in the face. The the sql statement will be more like active record and use ? for the value locations. The syntax for this statement comes from postgres. <br><br> Yah we made a statement.. but can we use it?<BR><br><br> <code> SpaceGhost.find(:ps,:key=>"Pstatement1",:conditions=>[100]) <br> SpaceGhost.find_by_ps(:key=>"Pstatement1",:conditions=>[15])<br>
I think its simple and sweet. We are also working on "auto_magic" mode for people who do not know sql. This will high jack active records processes of creating the sql, and create statements, manage them and run them instead of the normal sql statements. I just want to make sure no one can say I discriminate against people who do not know sql. Also, I have yet to test what happens if you call the a statement with the wrong model. Currently we do not have the concept of this model owns this statement. We need a better handling of errors and more testing with Oracle. But as Nick always says "Its piratically done"
Komodo 4
I use Komodo IDE at work and at home I would always use Rad Rails. That was the case until I started running 64bit suse. Rad Rails will not work for me, and it is not worth my effort to figure it out. Also Komodo just released 4.0!
I was sad at first because I did not see a price for a personal edition. So I am just the demo version of 4.0 and I found they have a free version which they call Komodo Edit. Found here ! I have yet to install the free version but if its close enough to the pay version I am set.
There is a big list some where of the features for komodo but there are only a few things that draw me to this product. Number 1 with out a doubt is the toolbox. I store bits of repeatable code and make them hot keys. This makes writing unit test, change logs, formatted rdocs comments so easy. I have also created a package for the tool box that includes the common RoR actions like rad rails has. That can be downloaded from here. Number 2 is the tab complete. In 4.0 you have to turn on the tab complete because its bound with different keys. I think they did a good job at guessing which word I want. They have an open that strips the whitspaces at the end of lines which includes those killer windows ^M's.
It is missing a few things that rad rails has. One is the hot key to take you from the controller to the view and back again. The auto formatting for indentation is very nice because sometimes i get out of control.
I was sad at first because I did not see a price for a personal edition. So I am just the demo version of 4.0 and I found they have a free version which they call Komodo Edit. Found here ! I have yet to install the free version but if its close enough to the pay version I am set.
There is a big list some where of the features for komodo but there are only a few things that draw me to this product. Number 1 with out a doubt is the toolbox. I store bits of repeatable code and make them hot keys. This makes writing unit test, change logs, formatted rdocs comments so easy. I have also created a package for the tool box that includes the common RoR actions like rad rails has. That can be downloaded from here. Number 2 is the tab complete. In 4.0 you have to turn on the tab complete because its bound with different keys. I think they did a good job at guessing which word I want. They have an open that strips the whitspaces at the end of lines which includes those killer windows ^M's.
It is missing a few things that rad rails has. One is the hot key to take you from the controller to the view and back again. The auto formatting for indentation is very nice because sometimes i get out of control.
Ruby should
Ruby should have a ternary operator for <=>
Ex:
string would be eqaul to "is less".. I will start working on this..
Ex:
string = 1<=>2 ? "is less":"are equal":"is greater"
string would be eqaul to "is less".. I will start working on this..
Gem to rails plugin
I most recently wrote a class to interface with IMBD.com. This class requires a library called hpricot. I locally gem install hpricot and create my code. I then added my code to my website. Do to the fact that I use shared hosting I do not have the proper rights to gem install my needed libraries. I could open a ticket with my hosting company, and wait for someone to get to it. Nah, I don't want to do that. I decided to look in to creating a plugin from a gem.
As it turns out most of the time you can.
This what I did to create a plugin for hpricot
I tried to do this with rmagik but it did not work. So its not 100% but it can be done. I am working on a rake command, much like rake rails, to create plugins from gems.
As it turns out most of the time you can.
This what I did to create a plugin for hpricot
1)Make rails project
2)ruby script/gen plugin hpricot
3)Gem install
a)If you can on the server change the gem root to a
local file you have RW access to and gem install hpricot -i /path/you/can/write
b)If you can not gem install on the server do it locally,but
do not cross windows with linux (different flavors do not seem to matter
to much)
4) Locate the installed gem package
5) Copy lib and ext from the gem to ror/vendor/plugins/hpricot/
6) Should work like a champ.
I tried to do this with rmagik but it did not work. So its not 100% but it can be done. I am working on a rake command, much like rake rails, to create plugins from gems.
Us websites
I found a list of words that end in us. I do not know how complete my list was but thats not the point. I made a script to find 3,4 and 5 letter words that I could make sites out of and a list of words that I could create with sub-domains. Just like script.aculo.us For example hero.us (taken) would give me the words [lecherous , ocherous , treacherous]. The longest set of words I have is for ero.us (taken), cio.us (take). oro.us(taken), geo.us (taken)
I can not find a 3 letter .us that is not taken. Well not one with many words.
So it looks like I am very late to this party. I added the text file to my svn http://svn.stephenbeckeriv.com/code/other/ussites.txt
I can not find a 3 letter .us that is not taken. Well not one with many words.
So it looks like I am very late to this party. I added the text file to my svn http://svn.stephenbeckeriv.com/code/other/ussites.txt
Segfault4r
I just released segfault4r. I created a rubyforge page,raa page and made an announcement to ruby talk. I can only hope the lord and master of ruby can forgive me for my sin.
Example:
Example:
require "rubygems" require "segfault" Segfault.now puts "Forgive me matz."
Rails conf day 3
Another great day at rails conf 2007! Morgan did his talk to day and it went well. The Extra Action Marching Band stopped by for a quick demo. I just got back from the bar they were playing in and it was amazingly fun. I talked to lots of people today google recruiters, tyler from amazon, some people working with nasa, and other people. I am excited to host my own presentation. I have to have someone with a Mac convert it in to a PDF first. I will post that link when I have it. I have found 2 blogs about my talk and one on morgans (all here). In the blog about morgan's talk they called us .net developers. That cuts me deep. I have never been a .net programmer. I have done (in order of first appearance) C,Ruby,Python,and a little Java. After checking out the blogs make sure to look on flickr. I am looking forward to tomorrow's talks.
Starting the end
Looks like 100% chance of rain.. I wanted to talk about the speeches I saw yesterday. I went to two talks on virtualization which I liked because I run VMs. I am glad to see that having a server in a VM only has a performance reduction of 5-8% I will be looking in to creating ruby scripts to create and manage Xen images. Another talk was about creating scriptaculous effects. I did want to see more of the graphic engine and that kinda logic but it was very informative. One example that was shown was blinding left to right or right to left. I want to add this effect to the calender on my page and start to create more effects. I am surprised some one does not have the star wars text scrolling effect. It would be fun to do but I think i will start out a little easier. Time to get some food and walk in the rain.
End of the end
It was a lot of fun. I have not posted about the talks because you can get the slides yourself. I will be posting what I create because of the talks I have seen. I am just glad that I could speak in front of so many of my peers.
Ruby formatter
Everyone has a preference on formatting code. I have found that there are a lot of very adamant people when it comes to the formatting of code. Unless a developer is using a language, like Python, that requires you to indent code it is up to the developer to correctly indent. When working with a team of people of you can have someone who does not care as much for strict formatting and you can have some one who is a formatting extremist.
To resolve some issues with other team members I have created a simple ruby formatter. I call it simple because it only formats one line at a time. There is not looking around at different lines, no knowledge of location in an object, or even scoping. I would like to create a rbchecker that would be like Pychecker, but this project is not that advanced. There is no parsing of ruby syntax, instead I am just looking for simple patterns. I am sure I missed a pattern or two but I have the option to create a back of the original file. I suggest turning on the back up for the first time you run this.
After I finished up a few test i figured i would look around to see if there were any other formatters. Turns out there are a few but, sine I am already done... Why not release it anyways?
Right now it is mostly comments, and no i did not run the formatter on it's self. I have been testing it on other projects of mine and koders.com ruby files.
Where to get it:
My svn http://svn.stephenbeckeriv.com/code/ruby_formatter/
How to use it:
Required:
-f file/location/e.rb
This can also be called with commas
-f file/location/e.rb,file/location/q.rb,file/location/w.rb
I am willing to change this if someone has a better idea.
Optional:
-s # will set using spaces with the size of an indent being #, the default is a tab
-b will create a backup of the file that is filename+",bk."+#{Time.now} I suggest using this when you run it for the first time.
ruby simple_formatter.rb -f /home/sbecker/test_file.rb
The only output should be "Done!"
ruby simple_formatter.rb -s 2 -f /home/sbecker/test_file.rb
Same formatted file now with two spaces instead of tabs
ruby simple_formatter.rb -b -f /home/sbecker/test_file.rb
Creates formated file as test_file.rb and the original file test_file.bk.#{Time.now}
I will be updating the code as a find more things that break it. Right now its worked for 95% of the code i tested it on. The only thing i know it does not work with currently is one line do end statements.
To resolve some issues with other team members I have created a simple ruby formatter. I call it simple because it only formats one line at a time. There is not looking around at different lines, no knowledge of location in an object, or even scoping. I would like to create a rbchecker that would be like Pychecker, but this project is not that advanced. There is no parsing of ruby syntax, instead I am just looking for simple patterns. I am sure I missed a pattern or two but I have the option to create a back of the original file. I suggest turning on the back up for the first time you run this.
After I finished up a few test i figured i would look around to see if there were any other formatters. Turns out there are a few but, sine I am already done... Why not release it anyways?
Right now it is mostly comments, and no i did not run the formatter on it's self. I have been testing it on other projects of mine and koders.com ruby files.
Where to get it:
My svn http://svn.stephenbeckeriv.com/code/ruby_formatter/
How to use it:
Required:
-f file/location/e.rb
This can also be called with commas
-f file/location/e.rb,file/location/q.rb,file/location/w.rb
I am willing to change this if someone has a better idea.
Optional:
-s # will set using spaces with the size of an indent being #, the default is a tab
-b will create a backup of the file that is filename+",bk."+#{Time.now} I suggest using this when you run it for the first time.
ruby simple_formatter.rb -f /home/sbecker/test_file.rb
The only output should be "Done!"
ruby simple_formatter.rb -s 2 -f /home/sbecker/test_file.rb
Same formatted file now with two spaces instead of tabs
ruby simple_formatter.rb -b -f /home/sbecker/test_file.rb
Creates formated file as test_file.rb and the original file test_file.bk.#{Time.now}
I will be updating the code as a find more things that break it. Right now its worked for 95% of the code i tested it on. The only thing i know it does not work with currently is one line do end statements.
Ruby class mixin
Some times in ruby you want your mixin to be a class method. Search the google for ruby class mixin returned little help on the matter. Although, if you have never did a mixin before I suggest reading the pick axe book. This example should look familiar if you have read the book
Currently I can not think of a good example of when this would be used, but it can be done. I understand why this works, but I think there would be a better way to achieve this.
#!/usr/bin/env ruby module Debug def new_method() new() end end class Phonograph class << self include Debug end end class EightTrack class << self include Debug end end puts Phonograph.new_method puts EightTrack.new_method
Currently I can not think of a good example of when this would be used, but it can be done. I understand why this works, but I think there would be a better way to achieve this.
Ruby formatter 0-5-0
Good News Everybody! Ruby formatter has an raa site! But the real good news is Andrew Nutter-Upham submitted a patch that changes how the options are done!
I like this formatting a lot better. You can also do a --help (if you have RI installed) and it produces something (I do not have RI installed). I hope to stay active with this project and look forward to hearing from more people.
ruby [options] filelist
options:
-s # will change the indent to a space count of # per level
by default we space with 1 tab per level
-b # create a backup file
examples:
ruby simple_formatter.rb -s 3 -b /moo/cluck/cow.rb
runs with the indent of 3 spaces,creates a backup file, and formats moo/cluck/cow.rb
ruby simple_formatter.rb -s 31 /moo/cluck/cow.rb /beef/lo/mein.rb
formats both files with 31 spaces per level.
I like this formatting a lot better. You can also do a --help (if you have RI installed) and it produces something (I do not have RI installed). I hope to stay active with this project and look forward to hearing from more people.
Ruby formatter base
There are a few basic rules I have for the simple formatter.
1) Key words are followed by a space
So no if(true). I want if (true)
2) Comments will be aligned with the current depth
I tried not formatting the comments depth, but I did not like the results
3) Block comments and here docs(soon) are left unformatted
The begin and end lines will be aligned but the inside of the comment/doc will not be touched
Thats about it right now. A few problems problems I fixed was using words like definition, comments after statements, and the order in which I strip user data. The current code in svn fixes these problems.
I have a few debugging statements or a debug mode if you will. Just set DEBUG_ME to true and watch some logic unfold.
1) Key words are followed by a space
So no if(true). I want if (true)
2) Comments will be aligned with the current depth
I tried not formatting the comments depth, but I did not like the results
3) Block comments and here docs(soon) are left unformatted
The begin and end lines will be aligned but the inside of the comment/doc will not be touched
Thats about it right now. A few problems problems I fixed was using words like definition, comments after statements, and the order in which I strip user data. The current code in svn fixes these problems.
I have a few debugging statements or a debug mode if you will. Just set DEBUG_ME to true and watch some logic unfold.
Ruby formatter 0-6-0
I have been busy with the ruby formatter. I made the formatter a class. By class I mean I took the code from simple_formatter.rb and created some methods. I did this so people can create their own interfaces for the formatter.
Example:
I also added a test file that I will add code to if someone sends me a chunk of code that breaks the formatter
I also fixed a few thing:
Here docs are not formatted
while statements at the end of blocks work
checks for one-line rescue statements
better command line options
better debugging statments and use of profiler
I strip out all regex, strings and comments before checking ruby code
Next step? Find more code to break the formatter and improve performance.
Example:
format_me = Formatter.new ruby_code =<<-RUBY def moo p ouch end RUBY puts format_me.format_string(ruby_code) format_me.format_file("/x/y/z.rb")
I also added a test file that I will add code to if someone sends me a chunk of code that breaks the formatter
I also fixed a few thing:
Here docs are not formatted
while statements at the end of blocks work
checks for one-line rescue statements
better command line options
better debugging statments and use of profiler
I strip out all regex, strings and comments before checking ruby code
Next step? Find more code to break the formatter and improve performance.
Ruby formatter
I have been using profile to find places to improve my code. I was looking at the profiler when I was in debug mode but, in debug mode I call methods more then once. After turning the profiler on and the debug statements off I was able to locate methods that were called more then once per line of formatted code. I also changed some if statements around and added constant regxs.
I saved a log my changes and put them up on the svn
I saved a log my changes and put them up on the svn
Action mailer recieving attchments
So I posted before about receiving voice mail and having populate on the internet. Well step one is checking the mail. As it turns out my email for my server is stored in a local dir. This is really good news because it appears that action mailer does not check email. First I need to add a receive method to my mail model:
Done! But you say it does nothing? I know. I hate changing code in models and then restarting the server. For now my logic will stay in the controller:
Now that I have access to the mail object and I can save the attached file I can now store the necessary data for my website.
For a list of methods on the TMail object try this website http://railsmanual.com/class/TMail::Mail
TMail
def receive(email)
email
end
Done! But you say it does nothing? I know. I hate changing code in models and then restarting the server. For now my logic will stay in the controller:
def test #open the local file and read it in. x=Notifier.receive(File.read("/home/giggles/realybadsite/1184671181.V9I760a7f6.popeyemail.com")) #at this point x is a TMail::Mail (links below) # x.attachments will be an array that has a TMail::Attachment which inherits from StringIO # This holds the attachment ruby/gems/1.8/gems/actionmailer-1.3.2/lib/action_mailer/vendor/tmail/attachment.rb # it also has original_filename and content_type File.open(Dir.pwd+"/tmp/"+x.attachments.first.original_filename,"w+"){|file| file << x.attachments.first.gets } end
Now that I have access to the mail object and I can save the attached file I can now store the necessary data for my website.
For a list of methods on the TMail object try this website http://railsmanual.com/class/TMail::Mail
TMail
Bad tag cloud
I made a bad tag cloud. Click the tag link at the top.
Lazy site syntax
I use the wonderful gem call syntax to convert my ruby code to pretty html blocks. My biggest problem with this is that I wrote my own CRUD interface. I have no fancy helpers for formatting or links. I thought that ruby code formatting would be a good first start. As I thought about it some more why should I need to convert the ruby when I am creating the post. Why can't my code convert it when it displays it? Lets try it..
All I need to do is wrap my ruby code in a code tag
I Could have had the converter code not be its own method, but I plan on using cvert else where in code. I also want to be able to set type attributes in my code tags that can change the converter.
All I need to do is wrap my ruby code in a code tag
if data =~ /<code>.*<\/code>/m data.gsub!(/<code>.*<\/code>/m){|code| code.gsub!(/<[\/]*code>/,""); cvert(code.strip)} end def cvert(string) convertor = Syntax::Convertors::HTML.for_syntax "ruby" return convertor.convert( string ) end
I Could have had the converter code not be its own method, but I plan on using cvert else where in code. I also want to be able to set type attributes in my code tags that can change the converter.
Formatter do fix
Someone emailed me today about my formatter being broken. I was incorrectly looking for do statements. For example I would only catch do |x| and not just do. I was also sending the string to the one liner check. These have been fixed and committed.
Rails upload file test
I wrote a rails service that can take a file as the posted params or data in the raw body. Like the good developer I am I wanted to create unit test for this functionally. Unsure how I could do this I search the Google for terms including but not limited to upload test file post rails with no good results. I did find fixture_file_upload which I now question what it really does.
I should have known the solution from odd test mishaps I have had before.
When you set params in the testing frame work they are passed as is.
For example:
This will set param[:file] to a Tempfile like a normal mutil-part form post would. If you did not find this out already Rails will not always return you a Tempfile for posted files. If the file is small enough it will be a StringIO object. If you are reading the contents they will function the same real_data = params[:file].read . If you plan on resending the data you might want to check the kind_of? on the object that was passed in.
anyway good to know stuff.
I should have known the solution from odd test mishaps I have had before.
When you set params in the testing frame work they are passed as is.
For example:
x = Tempfile.new() x.write = temp_file_data x.rewind post :service_x,:file=>x,value=>"x"
This will set param[:file] to a Tempfile like a normal mutil-part form post would. If you did not find this out already Rails will not always return you a Tempfile for posted files. If the file is small enough it will be a StringIO object. If you are reading the contents they will function the same real_data = params[:file].read . If you plan on resending the data you might want to check the kind_of? on the object that was passed in.
anyway good to know stuff.
Programming challenges
My roommate has a nice book called Programming Challenges. Today I started to work on the problems in this book. It's a simple way to keep the logic muscles in shape. I will not be putting the questions in the ruby files just the answers. I have the book from 2003 and it does not talk about using ruby for these but hey its what I know best. The link to svn has a %20 in it I am thinking about changing it.
Also, my search for an algorithm book is over. I am going to take "Introduction to Algorithms" ISBN 0-262-03293-7 and make all the sudo code and example in ruby. I was searching for a good book and I think I finally found one.
Also, my search for an algorithm book is over. I am going to take "Introduction to Algorithms" ISBN 0-262-03293-7 and make all the sudo code and example in ruby. I was searching for a good book and I think I finally found one.
Rjs partials
I was asked if one can do partial rjs files by a coworker recently. Some of our RJS code is becoming large and difficult to maintained. I figured why should RJS be different then the other templates. So I started up my test rails application. This is very filled with 1/2 crazy ideas and 1/2 unsure syntax examples. So I added two calls rjs1 and rjs2. I created a link_to_remote[1] on rjs1 pointing to rjs2.
I did nothing in rjs2 but created an rjs2.rjs file in side of my views directory. My rsj2.rjs file looks like
_moo.rjs looks like
What I do not understand is why firebug[2] only shows me the first replace call in the response.
What it is close to what I expect to but not complete. This is what I think I should see:
I am happy because this can help clean out view logic from the controller RJS calls, but I am also concerned that the improper display in firebug[3] can have a negative impact on developing JavaScript. I am also thinking that this can be done in a different way.
[1] http://api.rubyonrails.org/classes/ActionView/Helpers/PrototypeHelper.html [2] http://www.getfirebug.com/ [3] Firebug display issue
page.replace_html "one","one" page<<"#{render(:partial=>"moo")}"
_moo.rjs looks like
I understand that render will return the JavaScript of the rjs rendering. I also understand that I need to use page<< to append the JavaScript to the call like I would use <%= %> in an RHTML file. Would appear to work, but it seems off.page.replace_html "two","two"
What I do not understand is why firebug[2] only shows me the first replace call in the response.
try {
Element.update("one", "one");
} catch (e) { alert('RJS error:\n\n' + e.toString()); alert('Element.update(\"one\", \"one\");'); throw e }
What it is close to what I expect to but not complete. This is what I think I should see:
try {
Element.update("one", "one");
try {
Element.update("two", "two");
} catch (e) { alert('RJS error:\n\n' + e.toString()); alert('Element.update(\"two\", \"two\");'); throw e }
} catch (e) { alert('RJS error:\n\n' + e.toString()); alert('Element.update(\"one\", \"one\");'); throw e }
Note the nested try catch is only there because of the rjs debug setting. So I changed debug to false and restarted the server. I now see this in firebug:
Element.update("one", "one");
Element.update("two", "two");
I am happy because this can help clean out view logic from the controller RJS calls, but I am also concerned that the improper display in firebug[3] can have a negative impact on developing JavaScript. I am also thinking that this can be done in a different way.
[1] http://api.rubyonrails.org/classes/ActionView/Helpers/PrototypeHelper.html [2] http://www.getfirebug.com/ [3] Firebug display issue
Ruby documentation
I am looking for documentation on this functionality in ruby.
Clearly it returns the numeric value of 'g'. Why would I use this feature I can not find documentation about? Speed..104 == ?g #=> true
I am looking around my small set of books and the online documentation and have not found anything. I must admit I am unsure how to search for this.n = 5000000 => 5000000 Benchmark.bm do |x| x.report { 0.upto(n) do 104 == ?g end} x.report { 0.upto(n) do "g" == "g" end } x.report { 0.upto(n) do "g".eql?("g") end} end # user system total real # 1.270000 0.000000 1.270000 ( 1.330858) # 4.220000 0.010000 4.230000 ( 4.497720) # 4.190000 0.070000 4.260000 ( 5.206511)
Ruby documentation
I found the answer to my ruby code question. I found the documentation for assignment using *.
*x = one,two,three #=> x=[one,two,three] one,two,three = *x #=> this takes the x array and splits it out.
Rails rendering
No doubt any one looking at rails for speed will find that rendering sucks up a fair bit of time. I have two ideas to change the rendering system. The first is to remove the benchmarking code for production. I have no proof that this will make things faster. I am just sick of seeing it in my profiling log. The second idea I have is allowing the user of rails to change out rendering engines. For example using markaby instead of ERB or libxml(c lib) instead of XMLbuilder. I think using different libs could yield a faster render. If not with a c lib replacement then with another lib yet to be written.
Hotp
In my final days of work I have created a library for the HMAC-Based One-Time Password(HOTP) Algorithm. HOTP was created in hopes that a freely available encryption standard would generate more two-factor authentication devices. Looking around on the net you will find that there are a handful of devices that support HOTP. If you would like to read more about HOTP and its inner workings check out RFC 4226 (http://www.ietf.org/rfc/rfc4226.txt)
As for ruby and HOTP it is done. I hope to have a project up on RAA and rubyforge soon. A coworker helped me fill out the class, create a gem and correct a few implementation errors.
Example 1:
I think the RFC has you generate hotp numbers until you match the one that was passed in or you hit a max number of tries. This number would be inputed along with a personal pin and a unique id that you can related to the secret text for that card.
Example 2:
You get the point. I will post the links to the code when they are up.
As for ruby and HOTP it is done. I hope to have a project up on RAA and rubyforge soon. A coworker helped me fill out the class, create a gem and correct a few implementation errors.
Example 1:
#example numbers from the RFC require "HOTP" h = HOTP.new() h.secret = "12345678901234567890" h.count = "0" #can be a string or number. h.digits = 6 h.update #=> "755224" h.hotp #=> "755224"
I think the RFC has you generate hotp numbers until you match the one that was passed in or you hit a max number of tries. This number would be inputed along with a personal pin and a unique id that you can related to the secret text for that card.
Example 2:
#example numbers from the RFC require "HOTP" search_digits = ARGS[0] (0..10000).each{|counter| if HOTP::hotp("12345678901234567890",counter) == search_digits puts "you pressed the button #{counter} times!" break end }
You get the point. I will post the links to the code when they are up.
Hotp project
The HOTP code is released. You can gem install it because it is hosted on rubyforge. I also made an announcement on ruby talk. The Hotp class implements RFC 4226 in Ruby. (http://www.ietf.org/rfc/rfc4226.txt) HOTP is an HMAC-SHA1 based algorithm for one time passwords. The README has examples on how to use the library. We used the perl implantation and the RFC to write our test.
http://rubyforge.org/projects/hotp/ and http://raa.ruby-lang.org/project/hotp
http://rubyforge.org/projects/hotp/ and http://raa.ruby-lang.org/project/hotp
Login generator
I have posted about my ruby on rails generator that generates generators before. When I created this I did not know the generator code that well. Sadly this is still the case. I used generators for my one task saving me from created a nearly similar files over and over again. But Becker could you have not metaprogramed it? No each model had a common idea and interface but in the end the code was very different. I could have reduced the code and created a mixin for most of the methods, but it's a little late for that. Also I do not think I want to create a mixin for code that should be changing a lot.
Long story shorter I created some generators. I found my self creating another rails app for my websites and I am writing the login code over and over again. I thought why not generate? Then I thought why has someone not done this? But they have and, each of the generators work. They all have good attributes and bad ones but, none of them are 100% what I wanted. So I am going to make my own login generator. I will post it in my svn when its ok. I also realize the literature on generators is lacking. I also feel that the amount of generators that have been created is very low. It might be the most under used feature in RoR.
Long story shorter I created some generators. I found my self creating another rails app for my websites and I am writing the login code over and over again. I thought why not generate? Then I thought why has someone not done this? But they have and, each of the generators work. They all have good attributes and bad ones but, none of them are 100% what I wanted. So I am going to make my own login generator. I will post it in my svn when its ok. I also realize the literature on generators is lacking. I also feel that the amount of generators that have been created is very low. It might be the most under used feature in RoR.
Locomotive woes
After week one at Trusera I finally have my Mac configured to not use Locomotive. Locomotive is a wonderful little package of very common libraries for Rails on the Mac. It creates a sandbox of the libraries in some crazy tmp directory. I am confused on how they install Rmagick or ImageMagick with out really installing the libraries that are needed. I am concerned that the code I was using would not work out side the context of Locomotive even with the proper libraries and gems installed.
A fun problem to track down was during the booting of Rails via webrick in the Komodo IDE debugger I ran in to an error "constant RedHillConsulting::ForeignKeyMigration should be declared". I am not sure it was related to running webrick in Komodo, but I fixed this by requiring the RedHill libraries inside of the init.rb file for ForeignKeyMigration.
Other then that the other libraries installed very easily on the OSX 10.5 and I can use komodo's debugger.
A fun problem to track down was during the booting of Rails via webrick in the Komodo IDE debugger I ran in to an error "constant RedHillConsulting::ForeignKeyMigration should be declared". I am not sure it was related to running webrick in Komodo, but I fixed this by requiring the RedHill libraries inside of the init.rb file for ForeignKeyMigration.
Other then that the other libraries installed very easily on the OSX 10.5 and I can use komodo's debugger.
Firewatir
I never like to repeat the same tasks over and over again. I am sure this is why I am a programmer. With that said, I love to automate or streamline my daily/monthly processes. I have used Capistrano to create packages , copy files, and restart services. I have also created Selenium scripts to test website functionally.
My first week at work was mostly QA time and getting to know the code. One of tasks was to look in to automating test plans. I spent about a day trying to get Selenium working on my mac with no luck at all. So I started looking in to Watir(water). It looks like it is a very nice tool but, it is mainly a Windows based application. There is a Firefox only version called Firewatir(firewater) that works on window and OSX. I am unsure how I feel about the library api's but it is functional. Right now I find a lot of Watir docs written for the IE version, but I do not think the Firefox version is 100% yet.
It will be a slow learning process. Also I am learning a great deal about designing a website with automated testing in mind.
My first week at work was mostly QA time and getting to know the code. One of tasks was to look in to automating test plans. I spent about a day trying to get Selenium working on my mac with no luck at all. So I started looking in to Watir(water). It looks like it is a very nice tool but, it is mainly a Windows based application. There is a Firefox only version called Firewatir(firewater) that works on window and OSX. I am unsure how I feel about the library api's but it is functional. Right now I find a lot of Watir docs written for the IE version, but I do not think the Firefox version is 100% yet.
It will be a slow learning process. Also I am learning a great deal about designing a website with automated testing in mind.
Virtual server
I am using dreamhost's virtual servers. I am currently limited to 150mb of ram and a 150mhz cpu. This should be more then enough for my little web servers. I found after 10 days or so all my ram was being used. I think it is because of zombie dispatchers but I can never trouble shoot it to find out. I just added some tools that will allow me to see what is going on.
Oddly enough, I can not get top to run from rails. The top command will normally wait for input and keep updating frames. If you use -b -n1 it runs in "batch" mode and will cut out after one frame. This worked fine from irb and I was able to save the data to a string. In rails this fails, but I can ls, and ps. I normally only use ps to find a pid I want to kill. So I only know the -elf options and I grep for my data. After reading the man page for ps I found out I can out put the % of memory a pid is using. This was perfect. I will be able to tell exactly what the memory hog is.
Oddly enough, I can not get top to run from rails. The top command will normally wait for input and keep updating frames. If you use -b -n1 it runs in "batch" mode and will cut out after one frame. This worked fine from irb and I was able to save the data to a string. In rails this fails, but I can ls, and ps. I normally only use ps to find a pid I want to kill. So I only know the -elf options and I grep for my data. After reading the man page for ps I found out I can out put the % of memory a pid is using. This was perfect. I will be able to tell exactly what the memory hog is.
Woot and hpricot
I want to learn more about one of my favorite sites, Woot. It was the idea of my stat friend Drew, that he might be able to predict a woot off if he had the data about woot items. Easy enough to parse a website. After looking around woot for a few minutes I found that the blog holds all the data I needed. Lucky enough the data goes back to may of 2006.
Lets start..
Lets start..
Each blog page holds about 20 Woot items. It turns out that there are 58 pages of items. The Woot off days are days with more then one item.#!/usr/bin/env ruby require "rubygems" require "hpricot" require "open-uri" #my base blog url uri_base = "http://www.woot.com/Forums/Default.aspx?p=" #When parsing the woot pages I got an error and after a google search I found I needed to increase the hpricot buffer Hpricot.buffer_size = 262144
After this I create a CSV file with data. This allows me to re-parse the data later for anything.master_list = [] (0..58).each{|page| #open the page doc = Hpricot(open(uri_base+page.to_s)) #get all item rows row =(doc/'tr.itemRow') data =[] #create the date and title entries data = row.map{|row| ["#{(row/'div.saleMonth').text} #{(row/'div.saleDay').text} #{(row/'div.saleYear').text}","#{(row/'div.saleTitle/a').text}"] } #add the values of the data array not the array master_list.push(*data) #dont kill the server sleep(5) }
My code and data files are in my SVN.File.open("woot.data","w+"){|file| file<<"Date,Item\n" master_list.each{|woot_item| file.puts woot_item[DATE]+","+woot_item[ITEM].gsub(",","\,")+"\n" } }
Rails and soap4r
Recently at work I was tasked with making rails work with soap4r. There is a ticket on the soap4r trac[1] and in the Rails trac[2]. There are a wide range of solutions. I personally try to avoid changing core code if I can. With that in mind I have created two solutions.
In both solutions you need to not have soap4r install in the default gem folder. So gem install it in the Rails layout.
Solution 1:
Set the GEM_PATH in the rails project:
Setting gem_path after require rubygems is a great way to keep soap4r in one project. The gem library is to use once you find the documentation. For some reason it is not listed on the ruby-doc website but you should have a local version of the docs. In your environment.rb file add
Solution 2:
Use rubygem classes:
rubygems is just another ruby class to load other ruby classes. Its one of the reasons we are in this mess to begin with. The good news is I have searched for example code and found nothing useful
[1] http://dev.ctor.org/soap4r/ticket/433 [2] http://dev.rubyonrails.org/ticket/10001
In both solutions you need to not have soap4r install in the default gem folder. So gem install it in the Rails layout.
#since having soap4r installed will break all rail projects remove it. sudo gem uninstall soap4r sudo gem install --install-dir ~/rails_dir/vendor/gems soap4rYou can update the gem and remove it if needed. I see two negatives to this solution. One is that when I run IRB I do not have soap4r. This is solved by having it also installed in the normal gem dir on my dev box. Two there is going to be extra bandwidth with coping your rails code around. This can also be solved by installing it on your production boxes. If you have more then one prod box that could be a pain to remember to do.
Solution 1:
Set the GEM_PATH in the rails project:
Setting gem_path after require rubygems is a great way to keep soap4r in one project. The gem library is to use once you find the documentation. For some reason it is not listed on the ruby-doc website but you should have a local version of the docs. In your environment.rb file add
or the location you installed the gems to. Then gem install your little hard way.ENV['GEM_PATH'] = "#{File.dirname(__FILE__)}/../../../../gems" require "rubygems" Gem.clear_paths
Solution 2:
Use rubygem classes:
rubygems is just another ruby class to load other ruby classes. Its one of the reasons we are in this mess to begin with. The good news is I have searched for example code and found nothing useful
gems = Gem::SourceIndex.from_gems_in("~/local_gems")
spec = gems.search('soap4r').last
# straight from the gem requiring code base
# bin directory must come before library directories
spec.require_paths.unshift spec.bindir if spec.bindir
soap_path = ""
require_paths = spec.require_paths.map do |path|
tmp = File.join spec.full_gem_path, path
soap_path = tmp if tmp.include?("/lib")
tmp
end
# straight from the gem requiring code base
sitelibdir = Config::CONFIG['sitelibdir']
# gem directories must come after -I and ENV['RUBYLIB']
$LOAD_PATH.insert($:.index(sitelibdir), *require_paths)
#for some reason I needed to require path/soap/soap I am sure I can fix that
require File.join(path,"soap","soap")
[1] http://dev.ctor.org/soap4r/ticket/433 [2] http://dev.rubyonrails.org/ticket/10001
Sort vs sort!
I remembered reading a comment Matz made once about the ! methods. It went something like "! methods are destructive and should only be used when performance is needed". I found a benchmark example that show sort vs sort! speeds are depended on the garbage collection[1]. Today I was reading a chunk of ruby-prof code and found the comment.
I rewrote the code with a sort! in my head but wonder would it really have any affects on the time? A little context for test. I just used ruby-prof in an around filter in the application.rb file for a rails project. The html file that ruby-prof produces is not small and includes parts of the rails framework, mysql library, and core ruby code.
I just submitted a patch to the ruby prof ruby forge group[2]. It is not a big improvement over all. Using my numbers above the current GraphHtmlPrinter use 0.013572 just for sorting. It should now only take 0.004242. Saving the world most a second....
[1] http://www.ruby-doc.org/core/classes/Benchmark.html
[2] http://rubyforge.org/tracker/index.php?func=detail&aid=16364&group_id=1814&atid=7062
#from RubyProf::GraphHtmlPrinter def calculate_thread_times # Cache thread times since this is an expensive # operation with the required sorting @result.threads.each do |thread_id, methods| top = methods.sort.last thread_time = 0.01 thread_time = top.total_time if top.total_time > 0 @thread_times[thread_id] = thread_time end end
I rewrote the code with a sort! in my head but wonder would it really have any affects on the time? A little context for test. I just used ruby-prof in an around filter in the application.rb file for a rails project. The html file that ruby-prof produces is not small and includes parts of the rails framework, mysql library, and core ruby code.
Clearly not enough improvement to stop caching thread times but, now the methods are in order. We could stop caching and access the thread time likerequire "benchmark" def calculate_thread_times!(result) thread_times={} result.threads.each do |thread_id, methods| methods.sort! thread_time = methods.last.total_time > 0 ? methods.last.total_time : 0.01 thread_times[thread_id] = thread_time end end def calculate_thread_times(result) # Cache thread times since this is an expensive # operation with the required sorting thread_times ={} result.threads.each do |thread_id, methods| top = methods.sort.last thread_time = 0.01 thread_time = top.total_time if top.total_time > 0 thread_times[thread_id] = thread_time end end Benchmark.bm(12) do |test| test.report("no_bang:") do lresult = result calculate_thread_times(lresult) end test.report("bang:") do lresult = result calculate_thread_times!(lresult) end end # user system total real #no_bang: 0.010000 0.000000 0.010000 ( 0.006786) #bang: 0.000000 0.000000 0.000000 ( 0.004242)
Since the calculate_thread_times is called when you create a new RubyProf::GraphHtmlPrinter you should never need to call sort again. For example in the ERB template. I did find that using Benchmark.bmbm cause the real values to be swapped just like in the rdoc example[1].@result.threads[thread_id].last #I think
I just submitted a patch to the ruby prof ruby forge group[2]. It is not a big improvement over all. Using my numbers above the current GraphHtmlPrinter use 0.013572 just for sorting. It should now only take 0.004242. Saving the world most a second....
[1] http://www.ruby-doc.org/core/classes/Benchmark.html
[2] http://rubyforge.org/tracker/index.php?func=detail&aid=16364&group_id=1814&atid=7062
Large numbers of n
I found my self need to find the greatest number in an array of random numbers as fast as I can. Which sounds like a comp sci 101 problem visit every node and keep track of the highest number. I came up with a few different ways to do this with ruby's arrays and set up benchmarking.
Some might say its not fair. That I changed the out come by measuring it, but I had to do it. Thats right sorting a random array to find the largest value of that array is the quickest way I can find. I have also tried this with 60k and 600k arrays with the same relative speeds. Since Ruby 1.8 it has used the Schwartzian transform[1] for its sort method. Which is clearly faster then each but why is each_index twice as slow as each? I thought the size indexing test was slower because of the range object creation but I tried it with a precompiled range and it still was 8 seconds.
[1] http://en.wikipedia.org/wiki/Schwartzian_transform
And the winner is Schwartzian transform with max as a close second!n = 1000 marray = Array.new(6000) range = (0...6000) range.each{|i| marray[i]=rand(4000000)} require "benchmark" Benchmark.bmbm(1) do |test| test.report("Sort!:") do n.times{ array =Array.new(marray) array.sort! array.last } end test.report("max:") do n.times{ array =Array.new(marray) array.max } end test.report("Sort:") do n.times{ array =Array.new(marray) array.sort.last } end test.report("range index:") do n.times{ max=0 array =Array.new(marray) range.each{|x| max=array[x] if array[x]>max} } end test.report("index:") do n.times{ max=0 array =Array.new(marray) array.each_index{|x| max=array[x] if array[x]>max} } end test.report("size indexing:") do n.times{ max=0 array =Array.new(marray) (0...array.size).each{|x| max=array[x] if array[x]>max} } end test.report("each:") do n.times{ max=0 array =Array.new(marray) array.each{|x| max=x if x>max} } end test.report("uniq each:") do n.times{ max=0 array =Array.new(marray) array.uniq! array.each{|x| max=x if x>max} } end test.report("pop:") do n.times{ max=0 array =Array.new(marray) while x=array.pop max = x if x>max end } end end
#Rehearsal -------------------------------------------------- #Sort!: 1.380000 0.050000 1.430000 ( 1.462959) #max: 1.920000 0.010000 1.930000 ( 1.932106) #Sort: 1.380000 0.030000 1.410000 ( 1.409176) #range index: 8.060000 0.030000 8.090000 ( 8.121491) #index: 8.100000 0.070000 8.170000 ( 8.200113) #size indexing: 8.120000 0.060000 8.180000 ( 8.194519) #each: 4.840000 0.020000 4.860000 ( 4.884708) #uniq each: 7.140000 0.060000 7.200000 ( 7.203439) #pop: 5.590000 0.020000 5.610000 ( 5.611432) #---------------------------------------- total: 46.880000sec # user system total real #Sort!: 1.380000 0.040000 1.420000 ( 1.415057) #max: 1.920000 0.020000 1.940000 ( 1.927911) #Sort: 1.380000 0.040000 1.420000 ( 1.418863) #range index: 8.050000 0.020000 8.070000 ( 8.060964) #index: 8.070000 0.010000 8.080000 ( 8.079052) #size indexing: 8.070000 0.030000 8.100000 ( 8.085902) #each: 4.820000 0.010000 4.830000 ( 4.825778) #uniq each: 7.160000 0.070000 7.230000 ( 7.271699) #pop: 5.600000 0.030000 5.630000 ( 5.662141)
Some might say its not fair. That I changed the out come by measuring it, but I had to do it. Thats right sorting a random array to find the largest value of that array is the quickest way I can find. I have also tried this with 60k and 600k arrays with the same relative speeds. Since Ruby 1.8 it has used the Schwartzian transform[1] for its sort method. Which is clearly faster then each but why is each_index twice as slow as each? I thought the size indexing test was slower because of the range object creation but I tried it with a precompiled range and it still was 8 seconds.
[1] http://en.wikipedia.org/wiki/Schwartzian_transform
Iteration woes
My world has just been turned upside down. I need to start benchmarking some real data. I am using large arrays of random. Anyway, after trying to find the max number in array I found using the each method was slower then sorting. I then wondered if different types of iteration would be faster. So after some wings for breakfast at the Seattle airport I created another benchmark.
And the results?
If I left out any ways to iterate please let me know.
n = 10000 start_array = Array.new(6000) (0...6000).each{|i| start_array[i]=rand(4000000)} require "benchmark" Benchmark.bm(1) do |test| test.report("each:") do n.times{ start_array.each{|x| x } } end test.report("for:") do n.times{ for x in start_array x end } end test.report("each_index:") do n.times{ start_array.each_index{|i| start_array[i] } } end test.report("range each:") do n.times{ (0...start_array.size).each{|i| start_array[i] } } end test.report("each_with_index:") do n.times{ start_array.each_with_index{|x,i| x } } end end
And the results?
# user system total real #each: 14.360000 0.070000 14.430000 ( 14.516698) #for: 11.880000 0.040000 11.920000 ( 11.919000) #each_index: 44.280000 0.160000 44.440000 ( 44.451407) #range each: 44.760000 0.190000 44.950000 ( 45.039539) #each_with_index: 46.560000 0.320000 46.880000 ( 47.269472)
If I left out any ways to iterate please let me know.
Different includes
So there are many different ways to tell if an array includes an object. The results will change depending on where in the array the object is and if the object is in the array or not. A I once again used a random array to benchmark a few ways to do this.
I ran the test again but this time I inserted 5 at index 200. No shock they all cut out when they find the object they are looking for.
The results?n = 10000 start_array = Array.new(6000) (0...6000).each{|i| start_array[i]=rand(4000000)} require "benchmark" Benchmark.bm(1) do |test| test.report("include?:") do n.times{ start_array.include?(5) } end test.report("for loop:") do n.times{ found = false for x in start_array if x == 5 found = true break end end } end test.report("any:") do n.times{ start_array.any?{|x| x==5} } end test.report("find:") do n.times{ true if start_array.find{|x| x==5} } end test.report("select:") do n.times{ true if start_array.select{|x| x==5} } end end
# user system total real #select: 52.730000 0.400000 53.130000 ( 53.264748) #include?: 12.180000 0.020000 12.200000 ( 12.222207) #for loop: 44.630000 0.100000 44.730000 ( 44.863327) #any: 52.890000 0.450000 53.340000 ( 53.563788) #find: 43.590000 0.350000 43.940000 ( 43.999914)
I ran the test again but this time I inserted 5 at index 200. No shock they all cut out when they find the object they are looking for.
# user system total real #include?: 0.410000 0.010000 0.420000 ( 0.449498) #for loop: 1.500000 0.010000 1.510000 ( 1.545805) #any: 1.780000 0.000000 1.780000 ( 1.777178) #find: 1.770000 0.000000 1.770000 ( 1.773765) #select: 43.750000 0.400000 44.150000 ( 44.936202)
New imdb idea
So I was reading some comments from dzone[1] about my imdb "api". I do not like that the imdb site can change and I am SOL until I fix the code. For what I use my imdb class for it is not an issue but for some other people it most likely will be an issue. I have started work on creating migration scripts and data loaders for the downloaded version of IMDB[2]. I hope this project does not join my my dead projects list[3].
Happy new year...
[1] http://dzone.com/links/ruby_imdb_api.html
[2] http://www.imdb.com/interfaces
[3] http://stephenbeckeriv.com/dead_projects
Happy new year...
[1] http://dzone.com/links/ruby_imdb_api.html
[2] http://www.imdb.com/interfaces
[3] http://stephenbeckeriv.com/dead_projects
Futurama game
At my first seattle.rb meeting some people showed off the demos of Gosu[1], the ruby game enginey thingy. Anyway, with my passion for futurama being what it is I started on my first Futurama game! Also note this is the first time I am including a pic in a post.
With 147 lines of ruby I can fly the Planet express ship around the screen and fire lazers! Thats right lasers with a z. Anyway I plan on having some space pirate or space bandito ships flying around that you have to shoot. So far I just changed a demo graphics and added in bullets. I have an epic plan for my game. It will start like asteroids then change in to a Mario like side scroller then turn in to a defender like game. After that another Mario like level that ends at a pac-man like level that ends with space invaders.. If you know your Futurama its like the tales of interest show but a real game. I hope to have an svn repo soon. [1] http://code.google.com/p/gosu/
With 147 lines of ruby I can fly the Planet express ship around the screen and fire lazers! Thats right lasers with a z. Anyway I plan on having some space pirate or space bandito ships flying around that you have to shoot. So far I just changed a demo graphics and added in bullets. I have an epic plan for my game. It will start like asteroids then change in to a Mario like side scroller then turn in to a defender like game. After that another Mario like level that ends at a pac-man like level that ends with space invaders.. If you know your Futurama its like the tales of interest show but a real game. I hope to have an svn repo soon. [1] http://code.google.com/p/gosu/
Profanity filter of tomorrow
Announcing the Profanity Filter Of Tomorrow(PFOT) pronounce P-foot.
This is a web service call that will accept a posted text field and will return the text with the curses replaced or marked. Curses can include but not limited to obscenity, indecency and profanity. The service will be provided for one or many languages. Other services like spelling and grammar checking can also be added to the webservice call.
The service will accept and return different formatted responses including but not limited to Hyper Text Markup Language (HTML), JavaScript Object Notation (JSON), and Extensible Markup Language (XML).
Curse word dictionaries will be updated periodically by an aggregation of different sources including but not limited to humans and internet sources.
Why?
Although it is easy to program a simple white list to filter out profanity. There are a few problems with coding a white list. The first problem is the maintaining the list of words. Kids keep creating new ways to express those same old works. New slang, alternate spellings and fun elite(l33t) speak. A second problem is customization. I might want to filter some fields more or less strict or I might want to replace the bad words will fun silly words. I might want to replace the curses with curses from a different language. A third is foreign languages. Sure start filtering English and everyone will start cursing in French.
Stay tuned...
This is a web service call that will accept a posted text field and will return the text with the curses replaced or marked. Curses can include but not limited to obscenity, indecency and profanity. The service will be provided for one or many languages. Other services like spelling and grammar checking can also be added to the webservice call.
The service will accept and return different formatted responses including but not limited to Hyper Text Markup Language (HTML), JavaScript Object Notation (JSON), and Extensible Markup Language (XML).
Curse word dictionaries will be updated periodically by an aggregation of different sources including but not limited to humans and internet sources.
Why?
Although it is easy to program a simple white list to filter out profanity. There are a few problems with coding a white list. The first problem is the maintaining the list of words. Kids keep creating new ways to express those same old works. New slang, alternate spellings and fun elite(l33t) speak. A second problem is customization. I might want to filter some fields more or less strict or I might want to replace the bad words will fun silly words. I might want to replace the curses with curses from a different language. A third is foreign languages. Sure start filtering English and everyone will start cursing in French.
Stay tuned...
Cs forgotten
My memory was never the best. I have never worked on improving my memory because I can figure things out. If I do something enough it becomes second nature. The point is someone I work with was talking about Ternary search trees[1] and I had forgotten what they were. He started to describe what they are used for and how they basically work. I made the comment "it would be an interesting data structure in Ruby" which lead to the comment "Perl implements it with a hash of hashes." After the conversation the name of the tree left my mind and all I had was the perl comment and basic functionality. So I went to Cafe Presse[2] and start writing on a napkin while eating my Steak Frites. (I love those steak frites)
Basic functionally was defined as an interface that told me what words were in a dictionarythat was loaded, this is line for line what I wrote on the napkin
There was also a drawling of benders head and some flushed out hashes. It turns out this works well enough. Add in some load commands and different accessors and it should work well enough to get anyone started. The last bit of code that I added was an auto-complete part. I wrote this the next day at cafe vita over a double short americano(minus the comments)
It was fun. I liked that I didn't type a letter until after the code was written. I also like the implementation I did. The only thing I really knew going in to it was to store the data in a hash collection. After I wrote the code I googled around for ruby digital trie and found some sites[3][4] that made me feel good about how I wrote my code. Maybe this is the line of work I should be in after all.
[2] http://www.cafepresseseattle.com/pages/home.html
[3] http://www.ddj.com/windows/184410528
[4] http://www.rubyquiz.com/quiz103.html
Basic functionally was defined as an interface that told me what words were in a dictionarythat was loaded, this is line for line what I wrote on the napkin
@@top_hash = {} def create_hash(word) current_hash = @@top_hash word.strip! word.downcase! word.split(//).each{|letter| current_hash[letter] ||= {} current_hash = current_hash[letter] } current_hash["END"] = true end def is_word?(word) real_word = false word.downcase! word.strip! current_hash = @@top_hash (0...word.size).each{|i| current_hash = current_hash[word[i].chr] break unless current_hash real_word = true if ((i == word.size-1) && current_hash["END"]) } real_word end word = "gennosuke" create_hash(word) create_hash("ninja") create_hash("nine") pp @@top_hash pp is_word?(word) pp is_word?("cow")
There was also a drawling of benders head and some flushed out hashes. It turns out this works well enough. Add in some load commands and different accessors and it should work well enough to get anyone started. The last bit of code that I added was an auto-complete part. I wrote this the next day at cafe vita over a double short americano(minus the comments)
#get me to the end of the base then look for tails to add #there is improvements here if you can have compounded words def find_other_words(word_base) word_base.downcase! word_base.strip! current_hash = @@top_hash (0...word_base.size).each{|i| current_hash = current_hash[word_base[i].chr] break unless current_hash } find_tails(current_hash).collect{|tail| word_base+tail} end private #go all the way down the tails stack then build arrays on your way back up #can be cleaned up def find_tails(current_hash) if current_hash.kind_of?(Hash) current_tails = [] current_hash.each{|key,hash| if key != "END" find_tails(hash).collect{|tail| current_tails.push(key+tail)} else current_tails.push("") end } current_tails else [""]http://www.ddj.com/windows/184410528 end end
It was fun. I liked that I didn't type a letter until after the code was written. I also like the implementation I did. The only thing I really knew going in to it was to store the data in a hash collection. After I wrote the code I googled around for ruby digital trie and found some sites[3][4] that made me feel good about how I wrote my code. Maybe this is the line of work I should be in after all.
[2] http://www.cafepresseseattle.com/pages/home.html
[3] http://www.ddj.com/windows/184410528
[4] http://www.rubyquiz.com/quiz103.html
New month
Its new month but it's the same lame site. I kinda want to drop off the map and focus.
I am under the impression that the ruby singleton mixin does something like this.class PrivateNew class << self private :new end end
New organization
I want to start an organization that identifies useful common code libraries and their level of implementation status in the Ruby language. The organization will have a manifesto that will cover how libraries are tracked, picked and standards for implementation. There is still a lot to figure out and define.
Mod rails
The mod rails people have riddle on there site and I think I have the answer now.
blue = Apache (prefork MPM) + Passenger
red = Apache (worker MPM) + Passenger
yellow = Nginx Mongel
green = Nginx Thin
gray = Litespeed
purple = Apache (worker MPM) + Ruby Enterprise Edition + Passenger
This could be incorrect but I am hoping I got it right. I am excited as a dream host user to try out mod_rails!!
blue = Apache (prefork MPM) + Passenger
red = Apache (worker MPM) + Passenger
yellow = Nginx Mongel
green = Nginx Thin
gray = Litespeed
purple = Apache (worker MPM) + Ruby Enterprise Edition + Passenger
This could be incorrect but I am hoping I got it right. I am excited as a dream host user to try out mod_rails!!
Ruby formatter rails 2
I created a blank rails 2.0.1 project the other day and froze the gems to the vendor dir. I then used a script to gather all the .rb files in the vendor dir. I then ran my simple_formatter.rb script on them. Out of 654 files 611 did not throw a negative index error. This does not mean that all 611 were formatted correctly but it is a good sign that I am getting close to a complete formatter.
Ruby icalendar marshal
I am making a new application and I want to save Ruby Icalendar objects to a file. I am running ruby 1.8.6 (2008-03-03 patchlevel 114) [x86_64-linux] with icalendar gem 1.0.2.
True be told, I really only care about some of the data in the events. I am going to make an event class call becker_event that can take an icalendar event and allow it to be marshaled.file = File.open("Desktop/basic.ics") #=> #<File:Desktop/basic.ics> icals = Icalendar.parse(file) #=> lots of data Marshal.dump(icals) #=>boom! NoMethodError: Method Name: marshal_dump from /usr/lib64/ruby/gems/1.8/gems/icalendar-1.0.2/lib/icalendar/component.rb:413:in `method_missing'
Ruby post to blogger
While working on my newest project http://iblog.deathbyescalator.com/blog_import I hit a bit of a problem while posting to blogger. I did complete the project. It now uploads to blogger and wordpress but the google calendar import does not work.
Back to the problem at hand.. I found a few sites that looked promising. So I installed the GData-0.0.4 gem with no luck.
So after looking at the gdata gem and the blogger api I build a quick script to post a blog entry to blogger.
Full script at http://svn.stephenbeckeriv.com/code/gdata/blogger/
But the good parts: Auth
I got the HTTPS class from the GData lib. I like the idea. This call does nothing but authenticate with the blogger service and produce the needed header hash for calls later. This same method can be used for all the google apis.
Posting to the right blog
I have many blogs but want to post to just one of them. This method takes the body from a Http call i will show later. That call returns xml with all of the blog info. Links, author data, ids and other goodies. I do not think this is the best way to find the xml data I want. It is just one way that works.
Sending the post:
Simple enough. I send in an array that has the hash of entries to create. I build these methods for my needs. These can be easily adapted and changed. Hopefully a fully supported GData lib will raise from some where and save me from my one off scripts.
Back to the problem at hand.. I found a few sites that looked promising. So I installed the GData-0.0.4 gem with no luck.
So after looking at the gdata gem and the blogger api I build a quick script to post a blog entry to blogger.
Full script at http://svn.stephenbeckeriv.com/code/gdata/blogger/
But the good parts: Auth
module Net class HTTPS < HTTP def initialize(address, port = nil) super(address, port) self.use_ssl = true end end end GOOGLE_LOGIN_URL = URI.parse('https://www.google.com/accounts/ClientLogin') def get_gdata_headers(email,pass,service = 'blogger',source = 'gdata-ruby' ,url = 'www.blogger.com') headers = {} response = Net::HTTPS.post_form(GOOGLE_LOGIN_URL, {'Email' => email, 'Passwd' => pass, 'source' => source, 'service' => service }) raise "Not Authorized" unless response.kind_of? Net::HTTPSuccess headers['Content-Type'] = 'application/atom+xml' headers['Authorization'] = "GoogleLogin auth=#{response.body.split(/=/).last}" headers end
I got the HTTPS class from the GData lib. I like the idea. This call does nothing but authenticate with the blogger service and produce the needed header hash for calls later. This same method can be used for all the google apis.
Posting to the right blog
def find_blog_by_url_from_xml(blog_url,xml) xml_of_blogs = REXML::Document.new(xml) blogs = xml_of_blogs.each_element("/*/entry"){|z| z} urls = blogs.map{|e| # the first link is the full url that people can use. e.each_element("link"){|mr| mr}[0].attributes["href"] } index = urls.index(urls.select{|u| u.include?(blog_url)}.first) blogs[index] end
I have many blogs but want to post to just one of them. This method takes the body from a Http call i will show later. That call returns xml with all of the blog info. Links, author data, ids and other goodies. I do not think this is the best way to find the xml data I want. It is just one way that works.
Sending the post:
def post_xml_to_blog(base_url, headers, blog_url,post_array) conn = Net::HTTP.new(base_url, 80) # the http call that I was talking about.. list_of_blogs = conn.get("/feeds/default/blogs",headers) blog_xml = find_blog_by_url_from_xml(blog_url,list_of_blogs.body) blog_links = blog_xml.each_element("link"){|mr| mr} post_url = blog_links.select{|e| e.attributes["href"] =~ BLOGGER_FEED_REGEX }.first.attributes["href"] post_path = post_url.match(BLOGGER_FEED_REGEX).to_s response = Array.new post_array.each{|entry| response << conn.post(post_path, build_atom_entry_xml(entry).to_s, headers) } response end
Simple enough. I send in an array that has the hash of entries to create. I build these methods for my needs. These can be easily adapted and changed. Hopefully a fully supported GData lib will raise from some where and save me from my one off scripts.
Ruby code that makes me sad
I found some Ruby code that makes me sad.
I also found this..
This makes me wonder if the person who wrote this thought that String.new was filled with random bytes.class String def self.empty '' end end String.empty
I also found this..
I dont even think that this works for syntax sugar. Lets look at a few statements statements side by side and see how this concept looks.class Object def not_nil? !self.nil? end end "string".not_nil?
Give people sharp scissors and the will run with them. I really think the saying is give someone enough rope and they will hang themselves.if x puts "we have x" end if x.not_nil? puts "we have x" end # even better unless x puts "we do not have x" end unless x.not_nil? puts "We do not have x" end
Ruby underscore
Looking in to patching Hpricot for work and I found this bit of code
The example above is not good. It almost looks like the underscore is returning the right side of the assignment.
I first thought 'What is structure returning' but I was drawn to the _ (underscore) . I first thought that it was a pre-defined variables but I could not find it listed. So I fired up irb.tagname, _, attrs, sraw, _, _, _, eraw = structure[1]
The * in front of array just takes the contents of the array allow me to do multiple assignment. I would expect x to be 1, _ (underscore) would be 2, and y would be 3. When I display _ (underscore) it shows the the original array again. Then I print x which is 1 and then I print _ (underscore) again and it is now 1.x,_,y = *[1,2,3] # >] [1, 2, 3] [< _ # >] [1, 2, 3] [< x # >] 1 [< _ # >] 1 [<
The example above is not good. It almost looks like the underscore is returning the right side of the assignment.
Fun"asdf".sub("a","t") # "tsdf" _ # "tsdf"
Slacking at home
I am fixing up a few problems with iblog.deathbyescalator.com and I found my self repeating a chain of gsub statements. I did it for 8 statements then tested to make sure it fixed the issue I was seeing. I thought to my self how ugly.. but it only for a small app no one will every use. I would never let that kind of thing to fly at work, but since I am at home I feel like I can slack? Does the location that I am coding at matter for the quality of my code? Is it the fact that I am being paid at work and at home I am not matter? Maybe its the amount of people that will see my code that changes how well I code? Since I am the only one who will ever see the iblog code I feel less compelled to make it DRY or clean.
I dont know what the answer is but I didnt keep the chains of gsubs. I did the right thing and make it a method.
I dont know what the answer is but I didnt keep the chains of gsubs. I did the right thing and make it a method.
Icalendar blogging
I am calling icalendar blogging done. iblog.deathbyescalator.com. I don't have a better interface for it. It could easily be styled, add a logo, and give it a proper domain. This was just a flimsy 10 dollar idea.
Most interesting thing learned is how to use the icalendar library and how little you need to learn about atom to use it.
Most interesting thing learned is how to use the icalendar library and how little you need to learn about atom to use it.
Holidays and clockwork
I just bought a new domain http://deathtoholidays.com . I had no clue why I wanted to but this domain and was uncertain what to do with it.
But with uncanny timing a coworker was telling me about a library he is working on called Clockwork. After showing me a few examples I determined that I can use this in my new website. So I was going to create a Holidays library using Clockwork. Then it was purposed that I just added the library to Clockwork.
Good idea I says. So I am working on that and a new site..
You know between the book, the libraries, job, coffee roasting, boxing class ....
But with uncanny timing a coworker was telling me about a library he is working on called Clockwork. After showing me a few examples I determined that I can use this in my new website. So I was going to create a Holidays library using Clockwork. Then it was purposed that I just added the library to Clockwork.
Good idea I says. So I am working on that and a new site..
You know between the book, the libraries, job, coffee roasting, boxing class ....
Bad site
My site is still broken. I really need to take 3 minutes and fix it.
Moon man
Greetings moon man. I come in peace. I bring the Ruby MoonPhase module.
Ok so its not done done. But it is usable.
So where can you get this cheap port of c code? my svn repo. Gem coming soon
Ok so its not done done. But it is usable.
I would like to add this to the time and date ruby classes. So I say its not complete because we have big plans for it. Time.now.next_full_moonpp MoonPhase.moon_phase(Time.now+(-1*24*60*60)) #=> 0.999248104082693
So where can you get this cheap port of c code? my svn repo. Gem coming soon
Asset packer, rails and asset hosts
There is a nice feature in the action controller controller class that allows you to set an asset host. Its useful if you do not want to host your css, image, and javascript on the same server. Easy as pie to set
There is also a nice little plugin for rails call AssetPacker. It does some nice things like strip out white spaces and other things I should care about. But the point of this post is that these two concepts does not play well with each other.. Not yet at least..
Patch: asset_packager / lib / synthesis / asset_package_helper.rb
Current:
I do believe this kind of rewriting of a method on the fly makes it 3 times harder to debug but in the case it is 100% more useful.
[Update]
I failed to highlight the real issue. Asset host allows for 4 asset hosts. There is a method in Rails 2.0+ that will create asset0.host.com, asset1.host.com... and round robin the asset names. There are many sites on why this is good. The problem with the asset packer is that it redeclares compute_public_path. I am ok with that but it was leaving out a feature I wanted. I contacted sbecker on github and committed my patch to a fork. http://github.com/sbeckeriv/asset_packager/tree/master I am still new to github but I think I did everything ok. It turns out I can not run the rake test. It requires an extra rails project sbecker must have set up.
ActionController::Base.asset_host = "assets.mycoolapp.com" # or in the config scope conifg.action_controller.asset_host = "assets.mycoolapp.com"
There is also a nice little plugin for rails call AssetPacker. It does some nice things like strip out white spaces and other things I should care about. But the point of this post is that these two concepts does not play well with each other.. Not yet at least..
Patch: asset_packager / lib / synthesis / asset_package_helper.rb
Current:
private
# rewrite compute_public_path to allow us to not include the query string timestamp
# used by ActionView::Helpers::AssetTagHelper
def compute_public_path(source, dir, ext=nil, add_asset_id=true)
source = source.dup
source << ".#{ext}" if File.extname(source).blank? && ext
unless source =~ %r{^[-a-z]+://}
source = "/#{dir}/#{source}" unless source[0] == ?/
asset_id = rails_asset_id(source)
source << '?' + asset_id if defined?(RAILS_ROOT) and add_asset_id and not asset_id.blank?
source = "#{ActionController::Base.asset_host}#{@controller.request.relative_url_root}#{source}"
end
source
end
private
# rewrite compute_public_path to allow us to not include the query string timestamp
# used by ActionView::Helpers::AssetTagHelper
def compute_public_path(source, dir, ext=nil, add_asset_id=true)
source = source.dup
source << ".#{ext}" if File.extname(source).blank? && ext
unless source =~ %r{^[-a-z]+://}
source = "/#{dir}/#{source}" unless source[0] == ?/
asset_id = rails_asset_id(source)
source << '?' + asset_id if defined?(RAILS_ROOT) and add_asset_id and not asset_id.blank?
source = "#{asset_package_host(source)}#{@controller.request.relative_url_root}#{source}"
end
source
end
def asset_package_host(source)
if self.respond_to?(:compute_asset_host,true)
eval("def asset_package_host(source)\n compute_asset_host(source)\n end")
else
eval("def asset_package_host(source)\n {ActionController::Base.asset_host\n end")
end
end
I do believe this kind of rewriting of a method on the fly makes it 3 times harder to debug but in the case it is 100% more useful.
[Update]
I failed to highlight the real issue. Asset host allows for 4 asset hosts. There is a method in Rails 2.0+ that will create asset0.host.com, asset1.host.com... and round robin the asset names. There are many sites on why this is good. The problem with the asset packer is that it redeclares compute_public_path. I am ok with that but it was leaving out a feature I wanted. I contacted sbecker on github and committed my patch to a fork. http://github.com/sbeckeriv/asset_packager/tree/master I am still new to github but I think I did everything ok. It turns out I can not run the rake test. It requires an extra rails project sbecker must have set up.
Rash
I am no master of bash. I know even less commands in the windows shell. What I do know is ruby. So Rash is born. Bring my favorite shell commands to IRB. Now this is not a full Ruby Shell. There are a few projects out there doing this. This is just a port of some useful unix commands in to Ruby. I would love to support things like pipes. Prag Dave as a cool example using fibers to create pipes.
I have a few goals.
1) To be OS independent(as well as Ruby is) 1.5) Be pure ruby (1.8,1.9)
2) Blocks!
4) Be as close to the original command as possible
3) Grow it as I need it
I have a few goals.
1) To be OS independent(as well as Ruby is) 1.5) Be pure ruby (1.8,1.9)
2) Blocks!
4) Be as close to the original command as possible
3) Grow it as I need it
Doesn't look like much now and that is because its not much. It does make windows slightly more enjoyable for me. I can now also create other commands like remove_svn(true).require "bash.rb" #=> true include Rash #=> Object pwd #=> "C:/Users/sbeckeriv/private/ruby" cd ".." #=> "C:/Users/sbeckeriv/private" ls(nil,"TC") {|dirs| dirs.join(" | ")} #=> "TC/BGI | TC/BIN | TC/CLASSLIB | TC/DOC | TC/EXAMPLES | TC/FILELIST.DOC | TC/INCLUDE | TC/LIB | TC/README | TC/README.COM" sudo{rm "TC/README"}
Rpiratebay
I have started a ruby class for pirate bay. Using hpricot and my lack of a better idea I am crafting and interface to pirate bay. org. It has just started and is listed in my svn repo
There is also a pirate categories yaml file and a script to update it. The yaml file is just a ruby hash of the raw categories and organized hash. categories["Applications"]["UNIX"] == 303 . 303 is the number at the end of the browse url. I made this a config with an update script to cut down on scrapping times.
Here is a snippet of day one
There is also a pirate categories yaml file and a script to update it. The yaml file is just a ruby hash of the raw categories and organized hash. categories["Applications"]["UNIX"] == 303 . 303 is the number at the end of the browse url. I made this a config with an update script to cut down on scrapping times.
Here is a snippet of day one
pp PirateBay.get_browse_categories pp PirateBay::Categories::ALL_CATEGORIES pp PirateBay::Categories::TOP_LEVEL_CATEGORIES pp PirateBay::Categories::CATEGORIES_BY_NUMBER # returns an array of torrent objects recents = PirateBay.recent pp recents # sorts by file size pp recents.sort
Rails asset packager updated
I posted about a fix to asset packager to work with asset hots in rails. I contacted Scott Becker on Go get it!.
Github and praises
Git is the new hotness in CMS. I am not completely sold on its godly powers yet, but one thing I am really enjoying is github. After installing git via yast, setting up an account, creating a public key, and settings up the github account I was ready to roll.
After looking around and searching I found a few projects. You can download them with a simple command, follow them (update information kinda like twitter), and you can fork other peoples code. I dont know all the details of forking but I think its like svn copy. Either way you get your own copy of people's projects.
You can download make changes and push them back to the server. Then you send a pull request to the original repo (if you want to add the code back to the main branch). This has made it convenient for adding features or patching code. Not that this couldn't be done before. Github just kinda makes it centralized. It helps me on those days I am feeling less ambitious.
The whole point is that I have been using the Populator gem on a few projects. I have found that I often use a counter to track how many objects are created for a set. For example when I store order in the database. I created a fork, downloaded the code, patched it and pushed it back to github. Then I sent a pull request to ryanb. I am unsure what happens with pull requests. I think there is some code merging and fun-ness that happens. Either way it was easy and fast to do.
After looking around and searching I found a few projects. You can download them with a simple command, follow them (update information kinda like twitter), and you can fork other peoples code. I dont know all the details of forking but I think its like svn copy. Either way you get your own copy of people's projects.
You can download make changes and push them back to the server. Then you send a pull request to the original repo (if you want to add the code back to the main branch). This has made it convenient for adding features or patching code. Not that this couldn't be done before. Github just kinda makes it centralized. It helps me on those days I am feeling less ambitious.
The whole point is that I have been using the Populator gem on a few projects. I have found that I often use a counter to track how many objects are created for a set. For example when I store order in the database. I created a fork, downloaded the code, patched it and pushed it back to github. Then I sent a pull request to ryanb. I am unsure what happens with pull requests. I think there is some code merging and fun-ness that happens. Either way it was easy and fast to do.
Becker says with xmpp4r
I saw the gem xmpp4r-simple some where recently and thought back to the days of raimbo. I made a ruby script that was a google bot and did some other cute little functions. Thankfully that code is lost to the ages. The good news is that XMPP is the new hotness. Thats really good cause the raimbo lib does not work any more.becker = BeckerSpeaks.new becker.speaks_with("rubytalk@gmail.com",__RUBYTALKPASS__) becker.accepts_all_listeners! becker.says("sbeckeriv@gmail.com","on")
There is a nice library called xmpp4r-simple. I found lots of create examples of using this and using this with googletalk. For some reason I could not get it to connect to gmail. So I looked in to the underlying lib xmpp4r. After getting it to work with googletalk I started making BeckerSpeaks. The key to getting xmpp4r to work with gmail/googletalk was passing in "talk.google.com" when calling connect on the jabber client. This is also in the comments of another website. I did not find the site in time.
I have a few ideas for bots. 1) create a bot that informs me when a new url is tagged with ruby in delicious.com . 2) A bot that lets me know when a new thread is started on ruby talk 2) a library that sends me stacktraces from rails 2) A bot that does tasks 2) One that sends me an IM when I break a build 2) woot off bot 2) some kind of annoying bot to nag me about things. Lots of bots that kinda all do the same things. Just different conditions.
Xmpp4r jabber bots
I have started work on a jabber bot using xmpp4r and eventmachine. Its not much of a bot yet. Its more of a queuing system. People can subscribe to different queues ran off the same bot. For example rubytalk@gmail.com us currently sending out the new links in delicious tagged with ruby or rails. Invite rubytalk@gmail.com to chat and type show queues. Then type join:#{queue name} EX join:ruby rss. You get a little welcome message letting you know everything is ok. To leave the queue type leave:#{name}.
There is lots of work to do. I have the message manager creating and sending messages pretty well. I am unsure how I to flexibly handle user input. I think there is a way to centralize some of the standard logic and still allow users to easily add in new functionality. My main running script is about 100 lines and 50 of which are for parsing input.
Things to add: A member object and logic around members, input to the queues, more types of queues, help menu system, configuration, queue member persistence. I hope to put my code some where soon. Until then if rubytalk@gmail.com is on-line send them a message.
Update: You can now translate words, latin:forever or italian:hello. I support Latin, Italian, Spanish, French, German and Portuguese. The dictionaries are kinda limited but the price was right. The Internet Dictionary Project is where I got the files.
There is lots of work to do. I have the message manager creating and sending messages pretty well. I am unsure how I to flexibly handle user input. I think there is a way to centralize some of the standard logic and still allow users to easily add in new functionality. My main running script is about 100 lines and 50 of which are for parsing input.
Things to add: A member object and logic around members, input to the queues, more types of queues, help menu system, configuration, queue member persistence. I hope to put my code some where soon. Until then if rubytalk@gmail.com is on-line send them a message.
Update: You can now translate words, latin:forever or italian:hello. I support Latin, Italian, Spanish, French, German and Portuguese. The dictionaries are kinda limited but the price was right. The Internet Dictionary Project is where I got the files.
Bittorrent fun
I was looking around for a pure ruby bit torrent lib. I found rubytorrent but the ruby forge page was not working. I would like to use event machine to create a script to download torrent files in ruby. I have started by parsing torrent files.
After reading wiki pages, c and python code I wrote a class to decode the Bencoded files. The protocol website was down. The bencode breaks down to this. l means array, d means hash, #s means bits, and i means integer. They all end with an e. As the wiki example shows list can be nested and so can hashes..
Anyway, I have the file object setup.
Its in my svn repo http://svn.stephenbeckeriv.com/code/bittorrent/torrent_file.rb
After reading wiki pages, c and python code I wrote a class to decode the Bencoded files. The protocol website was down. The bencode breaks down to this. l means array, d means hash, #s means bits, and i means integer. They all end with an e. As the wiki example shows list can be nested and so can hashes..
I am not really happy with the decode code. I think it can be done much betterTorrentFile::Bencode.decode("l4:spam4:eggsl4:spam4:eggsei67ee") #=>["spam", "eggs", ["spam", "eggs"],67]
Anyway, I have the file object setup.
Step one down...tf = TorrentFile.new(@test_file) tf.name #=> name of the file or directory for the files tf.is_single_file? #=> true if there is no files key in the info hash
Its in my svn repo http://svn.stephenbeckeriv.com/code/bittorrent/torrent_file.rb
Palindromes
I was asked to check a file for palindromes. Here is the code I wrote.
# returns an array of matching palindromes in the file # the words are grouped in arrays # # [['god','dog'],['act','cat']] # def find_palindromes(file) if !file || !File.exist?(file) || !File.readable?(file) raise "Trouble with the file" end words = File.open(file,"r").read words = words.split(/\s/) words.uniq! palindrome = Hash.new(Array.new) for word in words palpalindrome_word = word,split.sort.join # you can downcase and remove chars here palindrome[palindrome_word] = palindrome[palindrome_word].push(word) end palindrome.values.reject{|words| words.size<2} end
Sort or in ruby
I am reading the perl cookbook and they are talking about some fun ways to use sort. I like that they are describing sort_by. I read some where that the concept was coded and posted in perl long ago. I wish they had a sort_by wrapper for it. anyway. I was reading that you can use an or statement in perl so if the first two are even sort it by a second value. I never thought of this in ruby.
array = [{:name=>"zack",:age=>30},{:name=>"becker",:age=>36},{:name=>"becker",:age=>24},{:name=>"morgan",:age=>35}] array.sort{|a,b| a[:name]<=>b[:name] || a[:age]<=>b[:age]} #=> [{:age=>24, :name=>"becker"}, {:age=>36, :name=>"becker"}, {:age=>35, :name=>"morgan"},{:age=>30, :name=>"zack"}]
Ruby and divide by zero
No more checking number.nan? || number.infinite?number = (1/0.to_f) #will not error number = 0 unless 1<=>number # will return nil if its infinity or NaN
Rvm
There is a new-ish gem called rvm that allows you to manage ruby vms and gems with a few simple commands. I just used it to unscrew my dreamhost server out of config hell with gems. In less then an hour i rebuild ruby 1.8.7 and ruby 1.9.1 and install most of the gems dreamhost gives a standard account.
I suggest looking at RVM very closely. I will never run another ruby box with out it.
I suggest looking at RVM very closely. I will never run another ruby box with out it.
When the state runs things
I live in yet another state that controls the liquor sales. PA is the other state. After talking with a coworker about some booze they showed me the state liquor store website. Turns out that they post the prices, stock and location of all the booze they carry.
After a night of scripting I have 2 scripts. The first one finds the brands and prices. The second one finds the quantities of each brand and what store they are at. Then it stores it in a mongodb (many reasons i am using mongo but that is not for here)
So after unscrewing dreamhost with rvm I finally have these scripts running on my server. But you say "big deal becker. what can you do with that data?". To which I say ..
The store at PORT ANGELES 1331 E FRONT STREET PT ANGELES WA 98362-4616 currently has 176 .75 liter bottles in stock of MACNAUGHTONS 36 MTH CDN TRAVELER which by the way is 80 proof! It is being sold at 12.45 including tax. Its not on sale currently...
I have more data but its getting late and my screens are ready to run all night.
After a night of scripting I have 2 scripts. The first one finds the brands and prices. The second one finds the quantities of each brand and what store they are at. Then it stores it in a mongodb (many reasons i am using mongo but that is not for here)
So after unscrewing dreamhost with rvm I finally have these scripts running on my server. But you say "big deal becker. what can you do with that data?". To which I say ..
The store at PORT ANGELES 1331 E FRONT STREET PT ANGELES WA 98362-4616 currently has 176 .75 liter bottles in stock of MACNAUGHTONS 36 MTH CDN TRAVELER which by the way is 80 proof! It is being sold at 12.45 including tax. Its not on sale currently...
I have more data but its getting late and my screens are ready to run all night.
When the state runs things ii
I picked up my liq site scraping again. I reworked my db using sql. I am going to use ruby 1.9.X and rails 3. It should be up soon.
When the state runs things iii
As a project I have been working on a website that scrapes data from the wa.liq state site. It was to include search, location based info, json data dump files for clients, and other fun features the state is to outdated to understand. I have had fun with selecting a brand type (Scotch) and a store (#28). I have a page which shows you the brands in stock but my page also shows you which brands they do not have that store but have at other stores. It also includes an approximate mile count to the closest store.
But as of Friday the state updated their website. Hooray! Not really. The site is still poor at best. The update comes with a new look. A look I do not like. There is a slight search improvement where you can search per brand type per city (not store like mine does. I was going to offer city). The usability of this also very poor. The find store now lacks the product you were looking for.
To top it all off they updated their system to some new version of ASP. It now includes value checking and javascript for form manipulations. It also holds the search part in an iframe so you have to see the ugly layout and buttons.
Long and short of it is that I am rewriting my parsers in mechanize. Its been fun to learn mechanize and its Javascript shortfalls. I thought all was lost when I read the Javascript and had mechanize mimic the code. I should be reparsing data by weekend's end. I take the update to the state site as a personal challenge. I will give the drinkers of Washington more features then they can dream of. When I am done here I am turning my sites to Oregon!
As long as the state is running things.
But as of Friday the state updated their website. Hooray! Not really. The site is still poor at best. The update comes with a new look. A look I do not like. There is a slight search improvement where you can search per brand type per city (not store like mine does. I was going to offer city). The usability of this also very poor. The find store now lacks the product you were looking for.
To top it all off they updated their system to some new version of ASP. It now includes value checking and javascript for form manipulations. It also holds the search part in an iframe so you have to see the ugly layout and buttons.
Long and short of it is that I am rewriting my parsers in mechanize. Its been fun to learn mechanize and its Javascript shortfalls. I thought all was lost when I read the Javascript and had mechanize mimic the code. I should be reparsing data by weekend's end. I take the update to the state site as a personal challenge. I will give the drinkers of Washington more features then they can dream of. When I am done here I am turning my sites to Oregon!
As long as the state is running things.
Food safety maps
I am working on a new project. A coworker asked why no one has mapped food inspections? I thought it was a good question. I found the data source for King county and put it on a google map. I added the project to GitHub. As soon as I figure how to host elasticsearch I will start linking it.
I think this project is a dead simple way to host and serve map data to google maps. I use elastic search as the back end, Sinatra for the server, Tire(ruby lib to elastic search) and a few other rag tag libs. It's not perfect but its a start.
I think this project is a dead simple way to host and serve map data to google maps. I use elastic search as the back end, Sinatra for the server, Tire(ruby lib to elastic search) and a few other rag tag libs. It's not perfect but its a start.
