Thursday, 12 April 2018

script - Simplest setup on Windows to send HTTP request, get and apply regex to response, and write match(es) to output file


I'd like to write a script that records the size of the close votes review queue on SO (currently ~95.5k), polling just a few times an hour, so I could plot a general trend. I know what I'm going to do regarding the parsing, i.e. given the following part of the HTTP response,


95.5k


I'd apply the regex



and split by \D and implode the array to leave only numbers, or something similar. (Yes, The Pony, He Comes, but this is a quick-and-dirty job during which I don't expect Stack Overflow's HTML to change.)


I don't currently have a UNIX / Linux setup, else I'd throw something together using cron, cURL, and Perl (or sed or awk if I'm feeling brave enough). What's the easiest way to do this on Windows? Is there some utility that's built to do this? I'm willing to install Cygwin and such software if it's indeed the easiest way (say, compared to writing batch scripts), but I'm hoping for some program into which I can supply a URL and regex and be on my way.



Answer



Actually, while waiting for someone to suggest a magical program to solve my every need, I decided to give Cygwin a shot, and found it was easier to do than I thought.


I simply



  1. downloaded Cygwin,

  2. made sure to check curl, cron, and cygrunsrv during installation,

  3. followed the steps described in this question (well, actually, I ran into some problems, but some Google searches suggested installing via cron-config with defaults, entering ntsec for the daemon, and inputting my Windows password, which worked),

  4. set up the following crontab:
    * * * * * /home/andrew/cron/get_cvrq_size.sh

  5. set up the following get_cvrq_size.sh:
    curl https://stackoverflow.com/review \
    | grep dashboard-num \
    | head -1 \
    | sed 's/^.*
    | sed 's/,//g' \
    | sed 's/^/'`date -Iseconds -u`',/' \
    >> /home/andrew/cron/cvrq_size.txt


and it's been working like a charm :-)


2013-11-25T20:05:01+0000,95583
2013-11-25T20:06:01+0000,95583
2013-11-25T20:07:01+0000,95583
2013-11-25T20:08:01+0000,95583
2013-11-25T20:09:02+0000,95589
2013-11-25T20:10:01+0000,95589
2013-11-25T20:11:01+0000,95587
2013-11-25T20:12:01+0000,95587
2013-11-25T20:13:01+0000,95586
2013-11-25T20:14:01+0000,95589
2013-11-25T20:15:01+0000,95587
2013-11-25T20:16:01+0000,95586
2013-11-25T20:17:01+0000,95585
2013-11-25T20:18:01+0000,95584
2013-11-25T20:19:01+0000,95596
2013-11-25T20:20:01+0000,95596
2013-11-25T20:21:01+0000,95596
2013-11-25T20:22:01+0000,95595
2013-11-25T20:23:01+0000,95595

No comments:

Post a Comment

Where does Skype save my contact's avatars in Linux?

I'm using Skype on Linux. Where can I find images cached by skype of my contact's avatars? Answer I wanted to get those Skype avat...