Two New Utilities to track player development

[badger ken]

I have written two new utilities to help in tracking player development. Let me know if you'd like a copy, if enough people are interested perhaps somebody will host it on their server:

1) I wrote a program which will "scrape" all 33 rosters (32 teams plus free agents) into a CSV file. the lines are of the form:
LastName,FirstName,Year,Team,Pos,Expr,Birthdate,College,Current,Future

a sample line:
Ferich,Kendyl,2008,Denver Broncos,LT,8,09/15/1977,"Indiana Univ., Pa.",82,83

you can now xcel/access this to your hearts content.
(nerd note - you will see that I quoted the college, and the college only, this is because some colleges, as in the example above, contain commas).

2) Suppose you run the program above every year. Take the CSV files so output and concatenate them together, and you now have an idea of how every player in the league has evolved over time. This can be handy when evaluating trade/free agents - is that 30/50 a 40/60 on the way down, or a 20/40 on the way up? I wrote a program which will do that, it prints out the top 20 busts, the top 20 booms,and then a color-coded printout for each payer. For example, here's the output for Kendyl Ferich from above, as I compared his 2008 to 2004 stats (the only two years for which I still had save-game files lying around):

Growth name Pos Expr team 2008 2004
37 Ferich, Kendyl LT 8 Denver Broncos 82/83 19/46

Yes, this guy went up 37 points, from a 19/46 to a 82/83 - he's the biggest boom in the league.
(the actual output looks a lot nicer, but a lot of the HTML, including the styles, got lost when I pasted it in here).

Just by way of comparison, here's the poor #1 bust:

Growth name Pos Expr team 2008 2004
-40 Moua, Lee RG 6 Free Agents 2/3 19/43

maybe he got injured?

anyways, let me know if you're interested in it.

[hitmanwa]

Definately interested here, how can I get a copy?

[fantastic flying froggies]

This sounds like a good thing. In fact, i would strongly recommend reposting your post in the general discussion forum, you will get a lot more looks and answers than here in the Strategy section, which is unfortunately lighly used.

I would also recommend PMing Suicane, he currently hosts most FOF utilities and I'm sure yours will interested many of us here.

[badger ken]

hitmanwa (and others) - just post with your email, or send me email (kfishkin@bigfoot.com)

froggies - thanks for the tip on the general forum, that forum seemed so clogged with non-FOF stuff that I was worried it would get lost in the noise.

I don't know how to "PM" somebody, could you enlighten me?

[hitmanwa]

Jeg19722003@yahoo.com

To send a PM click their name and use the send private message option.

Thanks for app,
Hit

[KWhit]

I would love a copy:

klwhit AT comcast DOT net

[Celeval]

I would love a copy:

klwhit AT comcast DOT net
Gotcha covered, KW:


Growth name Pos Expr team 2008 2007
19 Donaldson, Chris QB 2 Conyers Condors 14/22 57/81
28 Reagor, Randy RB 2 Conyers Condors 8/10 77/85

[gstelmack]

I think these utilities need to get up on a server somewhere. What happened to the utilities page that was being worked on?

Oh yes, and I'm interested:

gstelmack AT nc.rr.com

[jeronemitchell]

Definitely interested

jeronemitchell AT sbcglobal DOT net

[mhass]

anyways, let me know if you're interested in it.
Ummm.... Yes.

mhass AT yahoo DOT com

[fantastic flying froggies]

by the way, I'm also interested :)

patgaillard AT yahoo DOT com

[sovereignstar]

by the way, I'm also interested :)

patgaillard AT yahoo DOT com

Logic then tells me that you need to PM Suicane once you get the utility. He'll love the fact that you are giving something that he put work into some props.
:)

[hitmanwa]

I haven't recieved it yet...anyone else?

[mhass]

I haven't recieved it yet...anyone else?Yes. Haven't used it but I have it.

Edit: Thanks, KF by the way.

[cthomer5000]

I will host this on my webspace. I'll PM you my email address.

[KWhit]

Gotcha covered, KW:
That's not nice.

[Vinatieri for Prez]

I would love this utility. I'm in. How do I get it?

[cthomer5000]

This is now available at :

http://www.somethingwithanh.com/fofc/fofscrape.zip

[JeeberD]

Is it pretty easy to use?

[chinaski]

Im having problems with it. When it goes to scrape some teams, it tells me the file already exists, and asks if i want to overwrite it, and thats screwing up the whole process. Its the printer file that its trying to overwrite.

[badger ken]

Chinaski - I have had this at times also, sorry. It seems to be a Windows timing problem. I have found that if you let fofscrape finish, then exit, then re-launch it, then run it again, the problem often goes away. sorry about that :-(.

[chinaski]

Chinaski - I have had this at times also, sorry. It seems to be a Windows timing problem. I have found that if you let fofscrape finish, then exit, then re-launch it, then run it again, the problem often goes away. sorry about that :-(.
no trouble! :)

Ive tried a few times, by letting it finish then trying again, but it just wont work. I eventually get the output file, but it just has the 1st header line, but none of the scraped data.

Even with it not working (for me), this is excellent.. its the closest thing yet to having a true csv export for FOF.

[gstelmack]

Chinaski - I have had this at times also, sorry. It seems to be a Windows timing problem. I have found that if you let fofscrape finish, then exit, then re-launch it, then run it again, the problem often goes away. sorry about that :-(.
If the Windows printing turns out to be an issue, I'm pretty sure you could grab the data straight out of the screen. the TCY/FOF HTML export utility does this, and the technique isn't that difficult if the data is there. What language are you working in? If I sent C++ source, could you integrate that? If so, I could see about providing a routine to yank the data from the screen for you to dump. Have to slip that in among my work on an FBCB offline game viewer...

[badger ken]

great idea greg, after all my progam is basically just trying to do your program + "advance to next team".

I had thought that scraping the contents of the roster window wouldn't work, as the position and (more importantly) the current/future ratings are shown as pictures, not text - doing a "getitemtext"-type call will just return junk.

if you'd send me your code, that will be great, i will be delighted to integrate it in, happy to build on your code instead of re-inventing it. I would be happy to send you mine as well.

-ken

[fantastic flying froggies]

Don't you just love it when our utility gurus go along so well together? :)

[gstelmack]

great idea greg, after all my progam is basically just trying to do your program + "advance to next team".

I had thought that scraping the contents of the roster window wouldn't work, as the position and (more importantly) the current/future ratings are shown as pictures, not text - doing a "getitemtext"-type call will just return junk.

if you'd send me your code, that will be great, i will be delighted to integrate it in, happy to build on your code instead of re-inventing it. I would be happy to send you mine as well.

-ken
Bah, I'm failing miserably on my end. Just to clarify things, FOFExtractor uses an OCR algorithm to pull stuff out, which is required for the player reports because the data is simply not available to outside programs in any way I've found. But the grid displays appear to be, as at least the TCYHTMLExport program can grab them. But my quick attempts this morning are returning no data, so I'm not sure what I'm missing. You might want to get hold of the author of the TCY HTML Export utility and see if that code is available.

[chinaski]

Don't you just love it when our utility gurus go along so well together? :)

*sniff* its a beautiful sight!

[fantastic flying froggies]

*sniff* its a beautiful sight!
Aye, it brings a tear to my eyes...

[Whaler]

Newbie question here....

Aren't the ratings based on your scout? So, if you switch scouts (for example going from one who is EXCELLENT at QB to one who is FAIR at QB) does it have a large effect on the ratings?

This is probably more of a general question than one that relates to this utility.

Whaler

[cthomer5000]

This is now available at :

http://www.lovetohate.us/fofc/fofscrape.zip
The updated version is now available at the same link.

[badger ken]

I just sent out version 1.3, this version has the signal advantage of having had somebody other than me report that it works :-).

[jeronemitchell]

Got it working on my computer, and high recommendations to all. Now all we need is that companion version that tracks Current values... ;)

[chinaski]

hmm, this version wont work at all for me. It rewinds to the Free Agent list, pauses, then the scraper reports it cant find the team name.

Is there a newer version on its way up? Or does this new version have a revision date of 2/22?

[badger ken]

chinaski - are you using version 1.3? What you're reporting was a known bug in 1.2. To see which version, bring up the "about" menu, you get it by clicking on the little icon in the upper left of the dialog box.

[chinaski]

chinaski - are you using version 1.3? What you're reporting was a known bug in 1.2. To see which version, bring up the "about" menu, you get it by clicking on the little icon in the upper left of the dialog box.


aaah, thats the problem. The download link above is still for 1.2.... any chance we could get 1.3 up there?

thanks!
-Hank

[cthomer5000]

aaah, thats the problem. The download link above is still for 1.2.... any chance we could get 1.3 up there?

thanks!
-Hank 1.3 is now up.

http://www.lovetohate.us/fofc/fofscrape.zip

[chinaski]

Latest and greatest is working for me now.... but now that it works - I see there is an error in the csv output...

every once in a while there will be a line like this in the output csv...


Schwartz,Clarence,2009, Free Agents,TE,2,03/05/1986,"Washington State
Front Office Football 2004",7,26


Instead of a line break after Wash St., there should be that ascii square right before the "Front Office" part of that line.

It does that in about 30 random lines in the dump file.

[badger ken]

hmm, that is quite odd, the program isn't figuring out the college correctly.
Could I ask you to do this for me? (also, if others have similar bugs, a similar process would help):
a) see which team the player belongs to (in this case, free agents)
b) print the "scout overview" and "standard" versions of that teams roster to file - you will wind up with (say ) "scout.prn" and "standard.prn"
c) email them to me, with a description of what's incorrect in the CSV output.

I'm kfishkin AT bigfoot DOT com

[badger ken]

"Now all we need is that companion version that tracks Current values... "

if you're serious, that would take about 5 minutes to add in, would you like that facility?

[KWhit]

Yes. Absolutely.

[cthomer5000]

unfortunately this still doesn't work for me, it still closes out of the free agent roster window and opens Windows Media Player. http://dynamic2.gamespy.com/%7Efof/forums/images/smilies/confused.gif

[badger ken]

just released version 1.4:
1) fixes the bug reported above where sometimes a player in the .CSV files would have "<carriage return>Front Office Football 2004" embedded in their college name.

2) Responded to customer demand (2 posts :-)), and enhancd the skill tracker so that it tracks changes in present value as well as in future value.

cthomer - I feel you pain, but I can't replicate your bug, sorry - I'm stumped. Not sure if this will make you feel better or worse, but you're the only one with this problem :-(.

[Suicane75]

Added to the utility page. Jolly good work old chap.

[cthomer5000]

1.3 is now up.

http://www.lovetohate.us/fofc/fofscrape.zip
Versison 1.4 is now available at the same link.

[chinaski]

both links (suicanes and cthomers) are linked to a corrupt zip, when i download either one it says the zip is corrupted/incomplete. Either cthomer got it from Suicanes site, or Badger Ken is sending out a bad zip.

[cthomer5000]

i got it directly from ken.

[chinaski]

Ken sent out a fixed zip to everyone yesterday, hooray it works great too!

Ken - Any plans to dump the rest of the player data? i.e - height, weight, all the other positional ratings?

[badger ken]

re putting in height, weight, birthdate, etc. - umm, i guess i could, but does it really matter for anything? Just seemed like "chrome" to me. Please enlighten me :-)

[chinaski]

re putting in height, weight, birthdate, etc. - umm, i guess i could, but does it really matter for anything? Just seemed like "chrome" to me. Please enlighten me :-)

Im hoping to use this csv dump to write web apps, so having all the other player info would be spiffy, nothing major, but it still would be cool. Height, Weight arent that important to me, but the individual player ratings are... (current/future breakaway speed rating, current/future 3rd down passing - all that junk per position)

FOFExtractor already covers all of this and its probably a lot of tedious programming to add this FOFscrape, but FOFscrape cycles through all the teams and creates a master dump file, which i really like :)

So if youre bored and want to expand upon your fine app, this would be a useful addition :)

-Hank

[badger ken]

I found a free web hosting service, it has annoying pop-ups, sorry about that, but that's the price you pay for free hosting.
So anyways, you can now get FOFScrape at:
http://badgerken.50megs.com/files/fofscrape_small.zip

(The "_small" is because the web site only allows 250K files, so I had to take out the sample data files showing sample output, sorry about that).

The latest version is 1.4.1, I enhanced 1.4 to fix a bug where the program would crash when it had a player who didn't go to college (this is rare, but can happen).

Enjoy!

[bryce]

the link above wasn't working for me - if someone wouldn't mind emailing it to me at tutjer@gmail.com, i'd appreciate it.

thanks in advance.

[streetballer22]

Does this only work for FOF2K4?

[gstelmack]

the link above wasn't working for me - if someone wouldn't mind emailing it to me at tutjer@gmail.com, i'd appreciate it.

thanks in advance.
Not working for me either. I'm getting a bad permissions error.

[badger ken]

The kind folks at sourceforge.net have agreed to host fofscrape. This has the additional advantage, for all you FOF nerds, that I have posted the source code - you can see just how the program works, of course i would welcome bug fixed/enhancements.

I have put 3 files up at http://sourceforge.net/projects/fofscrape/ (click on the release to see the list of files you can download):

foscrape_small.zip - the release. Has the two executable programs and some documentation
fofscrape_source.zip - the C++ source code for fofscrape
playertrack_java_source.zip - the Java source for the "playertrack.bat" program.


enjoy everyone! Give it a few hours to percolate to sourceforge's servers, then we should all be good to go.

[alan_the_eagle]

I get an error message

"couldn't open standard roster file"

What does this mean?

[TurfToe]

I'm getting some Java errors. They are listed below. Is this a version issue or what?

Error message:

Exception in thread "main" java.lang.StringIndexOutOfBoundsException: String index out of range: -1
at java.lang.String.substring(Unknown Source)
at org.fofcentral.Player.getKey(Player.java:72)
at org.fofcentral.PlayerTrack.doit(PlayerTrack.java:370)
at org.fofcentral.PlayerTrack.main(PlayerTrack.java:425)

Thanks in advance.

[PhilSavage]

hmm, this version wont work at all for me. It rewinds to the Free Agent list, pauses, then the scraper reports it cant find the team name.I have this exact same problem...except I am using version 1.4.1!!

My situation:

FOF2004
Multiplayer league (I have tried scraping as participant and as commish)
We've changed the team names and city names
It's currently Preseason, week 4

Any thoughts?

[badger ken]

turftoe - this sounds like the bug that was fixed in 1.4.1, are you sure you are using that version? (See my post above about 'sourceforge', where the latest version is kept).

To help me debug this, could you email me the text file that you are feeding into playertrack?

PhilSavage - I can think of two things to do here:
1) remove your text-only printer, then add it back in, make sure that you use "Text/Generic" as the driver for the printer.
2) go to the temp directory that you established for FOFScrape, you should see two files ending in ".prn" called something like "scout.prn" and "std.prn" (can't remember exactly). If you see anything like that, could you email them to me? I can see where the bug is from that.

my email address again is kfishkin AT bigfoot DOT com

[PhilSavage]

Ken,

Thanks for the reply. I had the wrong printer driver.

It's all good now!

[TurfToe]

Thanks Ken! I must have had the wrong version. All I did was download from the link you posted and ran the batch file from there with my existing, scraped text files and it worked just fine.

Thanks again!

[mcsestretch]

badger_ken:

Fantastic utility! Thank you for making it.

Kudos to you!

Stretch

[NickSaban]

How do I set the default printer to FILE?

[sovereignstar]

How do I set the default printer to FILE?

First, make sure you've got the option to print to file (Instructions in the link below).

http://dynamic.gamespy.com/~fof/forums/showpost.php?p=288358&postcount=3

Next go to Start, Settings, Printers (maybe Control Panel before Printers) and right-click on the Text printer. There should then be an option to make it your default printer.

[NickSaban]

Ok, I set it up, and it looked like it scrolled up the team list to the free agents, but then it did some funky things. It opened Windows Media Player, muted my sound, and opened two instances each of Word and Excel.. As for the scraper program itself, it said "couldn't open standard roster file."

Help!

[jshipman2]

I'm getting the same problem.

[badger ken]

NickSaban, jshipman2 -
my, that _is_ bizarre! The program launches the print dialog by sending a <ctrl-P> keyboard command. Could it be that <ctrl-P> is bound on your computer to something else, like opening Media Player?

The message about "couldn't open standard roster file" means that your roster file either was never created (e.g. it opened Media Player instead for some reason), or that it was created but in a screwy format (e.g. your printer is of type FILE, but with the wrong driver - make sure that you specify a "generic text" driver).

[cthomer5000]

NickSaban, jshipman2 -
my, that _is_ bizarre! The program launches the print dialog by sending a <ctrl-p> keyboard command. Could it be that <ctrl-p> is bound on your computer to something else, like opening Media Player?
I have also had this problem with every version of the program, and it pains me because this is a utility i would reallllly like to use.

I examined Windows Media Player, and there is no assigned keyboard shortcut for it on my computer. What keyboard command does the program send to print?
</ctrl-p></ctrl-p>

[badger ken]

sorry guys, not only does this not happen on my machine, but i don't even have a machine anymore (my laptop died, i haven't re-installed FOF since then).

however, remember that the source is on sourceforge, have at it!

[Fonzie]

I finally got around to installing fofscrape, but I'm having the same "couldn't open standard roster file" problems as reported above. The program appears to open Windows Media Player before finally crapping out. Was anyone ever able to resolve this issue?

Oh, and yes, I'm using 1.4.1. This looks like a wonderful utility, but I just can't seem to be able to get it to work. Any help would be appreciated.

[chinaski]

I finally got around to installing fofscrape, but I'm having the same "couldn't open standard roster file" problems as reported above. The program appears to open Windows Media Player before finally crapping out. Was anyone ever able to resolve this issue?

Oh, and yes, I'm using 1.4.1. This looks like a wonderful utility, but I just can't seem to be able to get it to work. Any help would be appreciated.

I dont think theres any reason to use fofscrape now that FOFExtractor (http://gstelmack.com/fofextractor.htm) is on the scene.

[Fonzie]

I dont think theres any reason to use fofscrape now that FOFExtractor (http://gstelmack.com/fofextractor.htm) is on the scene.

I've got Extractor, but I haven't found an easy way to use it to track player ratings changes over time. Is it possible to do that with Extractor?

[sovereignstar]

I dont think theres any reason to use fofscrape now that FOFExtractor (http://gstelmack.com/fofextractor.htm) is on the scene.

Uh.. yeah there is.

[chinaski]

I've got Extractor, but I haven't found an easy way to use it to track player ratings changes over time. Is it possible to do that with Extractor?

ah yea, I forgot Extractor doesnt do player rating tracking. Scrape on! (once you can) :D