oarepo_oaipmh_harvester

OAI-PMH harvester for OARepo / Invenio RDM repositories.

User documentation (defining harvesters, running and scheduling harvests, writer options) is in the project README. This package documentation describes the internals.

Architecture

• oai_harvester - registry of harvesters (database model, service, REST resource /api/oai/harvest/harvesters). A harvester stores the OAI-PMH endpoint and the datastream definition (loader, transformers, writers).
• tasks - builds an invenio-vocabularies datastream (reader - transformers - writers) from a harvester and runs it as a Celery task.
• transformers - oai-import transformer converting the harvested XML with an import registered on the target model.
• writers - oai-service writer synchronizing the local record with the harvested one ("the harvest always wins") and tracking the result.
• oai_record - harvest records: one row per harvested OAI identifier with original / transformed data, local pid and errors (service, REST resource /api/oai/harvest/records, search index).
• jobs - per-harvester invenio-jobs job types.
• administration - invenio-administration views.
• permissions - permission policies and generators.
• cli - invenio oai commands, fixtures - oai_harvesters.yaml fixture.

Registration

The Flask extension OARepoOAIHarvesterExt is registered via invenio_base.apps / invenio_base.api_apps; its services, indexers and harvester jobs are registered in finalize_apps (invenio_base.finalize_app / api_finalize_app). Other entry points (pyproject.toml): database models and alembic migrations, search mappings, API blueprints, administration views, Celery tasks, the run_all_harvesters job, the oai CLI group and the oai_harvesters fixture.

Modules: