Miner

Generic scrape / block models (Django app agency.miner):

Django app agency.miner: generic web scrape operations and content blocks.

Host apps own domain-specific parsers and write into Scrap / Block.

Miner models: generic web scraping operations and extracted content blocks.

Domain-specific parsers (e.g. BOPV HTML) live in host apps and write into these models.

class agency.miner.models.Scrap(*args, **kwargs)[source]

Bases: Model

Database table: miner_scrap

Web scraping operation model.

Parameters:

id (UUIDField) – Primary key: Id
url (URLField) – Url
created_at (DateTimeField) – Creado el
updated_at (DateTimeField) – Updated at
status (CharField) – Status
error (TextField) – Error
data (JSONField) – Data
headers (JSONField) – Headers
cookies (JSONField) – Cookies

Reverse relationships:

Parameters:: blocks (Reverse ForeignKey from Block) – All blocks of this scrap (related name of scrap)

id

Type: UUIDField

Primary key: Id

A wrapper for a deferred-loading field. When the value is read from this

url

Type: URLField

Url

A wrapper for a deferred-loading field. When the value is read from this

created_at

Type: DateTimeField

Creado el

A wrapper for a deferred-loading field. When the value is read from this

updated_at

Type: DateTimeField

Updated at

A wrapper for a deferred-loading field. When the value is read from this

status

Type: CharField

Status

A wrapper for a deferred-loading field. When the value is read from this

error

Type: TextField

Error

A wrapper for a deferred-loading field. When the value is read from this

data

Type: JSONField

Data

A wrapper for a deferred-loading field. When the value is read from this

headers

Type: JSONField

Headers

A wrapper for a deferred-loading field. When the value is read from this

cookies

Type: JSONField

Cookies

A wrapper for a deferred-loading field. When the value is read from this

exception DoesNotExist: Bases: ObjectDoesNotExist

exception MultipleObjectsReturned: Bases: MultipleObjectsReturned

blocks

Type: Reverse ForeignKey from Block

All blocks of this scrap (related name of scrap)

Accessor to the related objects manager on the reverse side of a many-to-one relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

Parent.children is a ReverseManyToOneDescriptor instance.

Most of the implementation is delegated to a dynamically defined manager

get_next_by_created_at(*, field=<django.db.models.DateTimeField: created_at>, is_next=True, **kwargs): Finds next instance based on created_at. See get_next_by_FOO() for more information.

get_next_by_updated_at(*, field=<django.db.models.DateTimeField: updated_at>, is_next=True, **kwargs): Finds next instance based on updated_at. See get_next_by_FOO() for more information.

get_previous_by_created_at(*, field=<django.db.models.DateTimeField: created_at>, is_next=False, **kwargs): Finds previous instance based on created_at. See get_previous_by_FOO() for more information.

get_previous_by_updated_at(*, field=<django.db.models.DateTimeField: updated_at>, is_next=False, **kwargs): Finds previous instance based on updated_at. See get_previous_by_FOO() for more information.

objects = <django.db.models.Manager object>

class agency.miner.models.Block(*args, **kwargs)[source]

Bases: Model

Database table: miner_block

Content block extracted from a scraped page.

Parameters:

id (UUIDField) – Primary key: Id
content (TextField) – Contenido
metadata (JSONField) – Metadatos
created_at (DateTimeField) – Creado el
updated_at (DateTimeField) – Updated at

Relationship fields:

Parameters:: scrap (ForeignKey to Scrap) – Scrap (related name: blocks)

id

Type: UUIDField

Primary key: Id

A wrapper for a deferred-loading field. When the value is read from this

scrap

Type: ForeignKey to Scrap

Scrap (related name: blocks)

Accessor to the related object on the forward side of a many-to-one or one-to-one (via ForwardOneToOneDescriptor subclass) relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

content

Type: TextField

Contenido

A wrapper for a deferred-loading field. When the value is read from this

metadata

Type: JSONField

Metadatos

A wrapper for a deferred-loading field. When the value is read from this

created_at

Type: DateTimeField

Creado el

A wrapper for a deferred-loading field. When the value is read from this

updated_at

Type: DateTimeField

Updated at

A wrapper for a deferred-loading field. When the value is read from this

exception DoesNotExist: Bases: ObjectDoesNotExist

exception MultipleObjectsReturned: Bases: MultipleObjectsReturned

get_next_by_created_at(*, field=<django.db.models.DateTimeField: created_at>, is_next=True, **kwargs): Finds next instance based on created_at. See get_next_by_FOO() for more information.

get_next_by_updated_at(*, field=<django.db.models.DateTimeField: updated_at>, is_next=True, **kwargs): Finds next instance based on updated_at. See get_next_by_FOO() for more information.

get_previous_by_created_at(*, field=<django.db.models.DateTimeField: created_at>, is_next=False, **kwargs): Finds previous instance based on created_at. See get_previous_by_FOO() for more information.

get_previous_by_updated_at(*, field=<django.db.models.DateTimeField: updated_at>, is_next=False, **kwargs): Finds previous instance based on updated_at. See get_previous_by_FOO() for more information.

objects = <django.db.models.Manager object>

scrap_id: Internal field, use scrap instead.