4

TL; DR;

Is generating automated requests to a government site illegal in some way?

There is a government site (EU/Ireland) that provides a WebChat service as a contact option. When the chat is active there is a green button, otherwise, it is grey. It is almost impossible to find the green button (although sometimes it is green) so I decided to test the real availability of this service.

My idea was to create a script that accesses the website, checks the status of the button and then logs it. My goal was not to DDoS the server or something similar so I thought a request every minute will suffice for getting some statistics.

Could this be considered illegal? I know it might depend on each country's laws but if someone in the field could provide some insight or directions is really appreciated.

Even more Details:

The website is always up even if the chat is not available, so it is not only testing an HTTP request. What I found is that the HTML is loaded without the "button" (it's not a button tag, it's actually a clickable image), then a javascript script connects to the chat service and depending on the status, it loads the "button" to access the chat or to notify it is unavailable. This is an <img> tag. What I do is read the "alt" property description where the status can be identified. Then I put that in a cron job, log the time and the status. Finally, at the end of the day, I summarise the data into a chart showing the percentage of time the chat was really active. The purpose of this experiment is to prove that the customer service is awful with actual data.

Improve this question

1 Answer 1

Reset to default
14

What you are doing is commonly referred to "web scrapers" and they are legal in the EU.

What you cannot do is extract personal data. Since the data you are aggregating is non-personal data, whether or not a chat button is available, it should be fine.

EDIT

As some of the commenters said - it's legal, but many websites detect scrapping. To (try to) avoid being blocked by the server, make it act human. Something like. I check once every 15 minutes with 3 minutes +/- is probably enough. That also is probably what a human would do using the website so it should strengthen your argument the website is unavailable.

Improve this answer
4
  • Thank you for your answer. I have provided more details about the scrapping process and the usage of the data
    – gustavovelascoh
    Commented Apr 14, 2022 at 16:22
  • 7
    Some servers have software to detect web scraping and block the offending user. To minimize the risk of getting blocked, ensure your requests are slow enough that they look like a human could be sending them manually. Randomizing the delay between page accesses helps as well.
    – bta
    Commented Apr 15, 2022 at 1:24
  • 4
    Randomising the delay may also improve the scientific validity of your result, by avoiding your sampling falling into lockstep with any pattern in the availability of the button. That's not what you are asking about, but it's a nice little bonus.
    – Josiah
    Commented Apr 15, 2022 at 11:10
  • 1
    It should be noted that 1 request a minute (average) is web scraping, but 1 request per microsecond is a DoS attack (DoS if you're doing it from a single computer/network, DDoS if from multiple computers/networks). There is a line somewhere in between, and some information about who decides where it is would be interesting.
    – GuilleOjeda
    Commented Apr 15, 2022 at 20:23

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .