PHPFixing
  • Privacy Policy
  • TOS
  • Ask Question
  • Contact Us
  • Home
  • PHP
  • Programming
  • SQL Injection
  • Web3.0

Sunday, September 4, 2022

[FIXED] How to access a website with a non-inspectable login pop-up

 September 04, 2022     authentication, beautifulsoup, python, web-scraping     No comments   

Issue

I am trying to write a program to scrape data from this website: https://www.dmr.nd.gov/oilgas/feeservices/getscoutticket.asp

The issue is when you go to that link, a login pop-up appears that is non-inspectable. There is only a blank html page behind it, and I cant figure out how to get the username and password(which I have) into the login box. I have tried using various post requests, but as the pop up doesn't have a separate URL and doesn't seem to be accessible from my current code, none of them are working

from bs4 import BeautifulSoup
import requests

url = 'https://www.dmr.nd.gov/oilgas/feeservices/getscoutticket.asp'
s = requests.Session()
payload = {
    'dbconnect': 'y',
    'entryPoint': 1001,
    'nomblogon': 0,
    'Username': 'username',
    'Password': 'password',
    }
s.post(url, data = payload)

site = s.get(url)
soup = BeautifulSoup(site.content, "html.parser")

Has anyone dealt with this type of login pop-up before? Is there any way to navigate past this using code?

I am currently writing in python, but java answers would also be appreciated.


Solution

That non-inspectable pop-up is Basic HTTP Authentication, and you handle it with something like this, in requests:

from requests.auth import HTTPBasicAuth

basic = HTTPBasicAuth('user', 'pass')
requests.get('https://www.dmr.nd.gov/oilgas/feeservices/getscoutticket.asp', auth=basic)

I suggest reading the requests documentation, which can be found at https://requests.readthedocs.io/en/latest/user/quickstart/



Answered By - platipus_on_fire
Answer Checked By - Robin (PHPFixing Admin)
  • Share This:  
  •  Facebook
  •  Twitter
  •  Stumble
  •  Digg
Newer Post Older Post Home

0 Comments:

Post a Comment

Note: Only a member of this blog may post a comment.

Total Pageviews

Featured Post

Why Learn PHP Programming

Why Learn PHP Programming A widely-used open source scripting language PHP is one of the most popular programming languages in the world. It...

Subscribe To

Posts
Atom
Posts
Comments
Atom
Comments

Copyright © PHPFixing