Topic of the 2020 Pyton Developer Online Technology Summit: Technology Implementation of the Crawler Framework and Experience Sharing with Module Applications

Article Directory 1. Preface 2. Some of the concepts we must know about Crawlers 2.1 Definition of a crawl 2.2 Legal risk of Crawlers 2.3 Understanding crawl types from crawl scenarios 2.4 Basic techniques and crawl framework for Crawlers 3. Data Grabbing Technology 3.1 Tencent NPC epidemic data d ...

Posted on Wed, 12 Feb 2020 18:00:13 -0800 by jumphopper

Data mining practice of financial websites

Article directory 1. real time data mining of sina finance stock 2. Dongfang fortune data mining 3. The actual battle of data mining in referee document network 4. Data mining of cninfo 1. Sina Financial Stock real-time data mining The previous post introduced the use of Selenium The code is as ...

Posted on Sat, 08 Feb 2020 05:19:04 -0800 by border20

Selenium Python Usage Tips

Book above and above: Selenium Python Usage Skills (1) Selenium Python Usage Skills (2) Handle waiting in different situations In elenium automated tests, a Web page may take some time to load, or you may want to see specific Web elements on the page before triggering the test code.In this case, you need to execute Explicit Wait, which is a p ...

Posted on Fri, 07 Feb 2020 17:54:40 -0800 by Dillenger

Pthon Batch Query Douban Book Score (source code attached to the tutorial)

stay Lazy Disk Shared high-score e-books are generated using python batch queries The regular bean-petal api is not allowed to be invoked, several searches reveal an interface https://book.douban.com/j/subject_suggest?q=book name Use this interface to get the url of the book on the bean petal Functions ...

Posted on Mon, 03 Feb 2020 18:21:05 -0800 by DuNuNuBatman

Python: Crawl daily epidemic data

Preface At present, every major platform, such as Tencent and today's headlines, will update the daily epidemic data, and their data sources are the same, mainly through the health care commission's official network. Taking the whole country, Hubei and Shanghai as examples, there are three websites: T ...

Posted on Sun, 02 Feb 2020 18:41:32 -0800 by maxat

Python 3 uses selenium + Chrome basic operation code

selenium is a third-party library of Python, which needs to be installed before use. But if you use anaconda, you can omit this step. Why? Bring your own, willful. Installation command: pip install selenium (1) Use selenium to open the specified website. Take Taobao for example. # -*- coding: utf-8 -*-"""Created on Wed Jul 25 10:12:39 2018@a ...

Posted on Fri, 31 Jan 2020 15:20:15 -0800 by Lerris

Using Python to download data on LAADS automatically and in batches

Catalog Preface Profile data download link Python+Selenium+ChromeDriver configuration Using Python+Selenium to call wget to download data Using Python+Selenium to call IDM to download data summary 1. Preface LAADS(https://ladsweb.modaps.eosdis.nasa.gov/ )It's a NASA data distribution website. On t ...

Posted on Wed, 29 Jan 2020 02:30:17 -0800 by Dax

Screenshot operation of java selenium

Article directory Selenium screenshot operation is a common WebUI operation. Here is a detailed introduction of how to implement screenshot operation and the whole process of selenium project failure screenshot Prerequisite The project is a maven project and requires the following dependency package ...

Posted on Tue, 28 Jan 2020 21:48:51 -0800 by Gimpy

Python notes: the use of Selenium library and the operation case combined with the Scrapy framework

Introduction to Selenium Library Selenium is an automatic testing tool, which can drive the browser to perform specific actions, such as click, drop-down, etc Selenium can obtain the source code of the page currently presented by the browser, so that it can be seen and crawled. It is very effective t ...

Posted on Fri, 17 Jan 2020 20:42:16 -0800 by gavinr98

Java+Selenium automated testing

Integration of Java+Selenium+TestNG automatic test framework 1. Simplified code Encapsulates a class for locating elements, of type ElementLocation package com.test; import org.openqa.selenium.By; import org.openqa.selenium.WebDriver; import java.util.concurrent.TimeUnit; /** * Call the same method every time you locate an element * Encapsul ...

Posted on Thu, 16 Jan 2020 08:03:30 -0800 by Nightslyr