3
$\begingroup$

It seems to me that we should at least collect some data to move our first polystats project forward. A little bit of data collection and doing some exploratory plots may really help with refining our project questions. I have spent the past two days looking around the API/data dump and am at a loss how to use them to extract the relevant data. Clearly I am out of my depth here!

Could someone with some experience help in downloading some data for this project? We could use this thread to discuss the specifics.

PS: Unlike other polystats project questions on the parent site I really do not have a statistics question here per se and hence I am posting on meta instead of the parent site.

$\endgroup$
11
  • $\begingroup$ Just as a note, on the data explorer you can only query 2000 items at a time, for an example odata.stackexchange.com/stackoverflow/q/11896 , so this is probably not a viable option. I can not download the torrent at the moment on my work network, but I believe this is the link, clearbits.net/get/1419-sept-2010.torrent $\endgroup$
    – Andy W
    Sep 30, 2010 at 14:28
  • $\begingroup$ I might do this over the weekend, if no one else gets to it first. $\endgroup$
    – Shane
    Sep 30, 2010 at 18:52
  • $\begingroup$ @andy Thanks for the info. I will see if I can make use of the torrent files. @shane Thanks. Assuming that you do find the time would you download all the data or some specific subset? $\endgroup$
    – svadali
    Oct 1, 2010 at 0:44
  • $\begingroup$ I finished downloading the torrent a few minutes ago. I am getting rid of the meta files and zipping it back up. I have yet to decide how I will disseminate via url but if you give me an hour or two I should be able to post a link. It will probably be about 500 megabytes. $\endgroup$
    – Andy W
    Oct 1, 2010 at 1:54
  • $\begingroup$ @Andy Great! Just out of curiosity what is the format of the file? Perhaps, the files can be hosted on the polystats wiki if that is fine with @shane. $\endgroup$
    – svadali
    Oct 1, 2010 at 2:06
  • $\begingroup$ They are in xml format (each site has its own xml file for the distinct tables), the original file downloaded from the torrent is compressed with 7z , It is 1 gigabyte. Still not sure exactly how I would either share that or the zip file I am compressing now. $\endgroup$
    – Andy W
    Oct 1, 2010 at 2:12
  • $\begingroup$ I apparently overestimated my compression softwares abilities. It will be easier for me to just disseminate the 7z file I downloaded from the torrent that is 1 gig in size. I have a proprietary software that could decompress this file type (WinZip), but is the 7z compression ok for other people? I can put this file on my Dropbox folder and publish a link. $\endgroup$
    – Andy W
    Oct 1, 2010 at 2:36
  • $\begingroup$ @Andy Please go ahead and post the link. If it does not work for some we can find out what to do later. Add the link as an answer to this question so that everyone can see it. Thanks for your help. $\endgroup$
    – svadali
    Oct 1, 2010 at 2:48
  • $\begingroup$ Here is the link for the 7z file of the entire data dump as of 9-10 dl.dropbox.com/u/3385251/Export-090110.7z , it is still uploading to Dropbox on my computer, so give it a few more hours before tring the link (about 4 am East Coast USA). It just gives a 404 error message if it is not uploaded yet. $\endgroup$
    – Andy W
    Oct 1, 2010 at 4:57
  • $\begingroup$ @Andy Are there any bandwidth issues for you if lots of downloads are made from your dropbox? $\endgroup$
    – svadali
    Oct 1, 2010 at 10:52
  • $\begingroup$ @Srikant I don't believe so. It is an online service, so it is not my personal server. I tested to make sure the link worked this morning but did not download the entire file. Let me know if the link does not work or the file is corrupted if anybody does download it (I'm blocked from the site and my own link while at work). $\endgroup$
    – Andy W
    Oct 1, 2010 at 12:41

1 Answer 1

Reset to default
2
$\begingroup$

Andy has downloaded the file and cleaned it up a bit. You can download the file at the link given below:

http://dl.dropbox.com/u/3385251/Export-090110.7z

I believe that the file format is xml and that it is a compressed file. Thanks Andy.

$\endgroup$

You must log in to answer this question.