Bulk Operations with FKAPI

Overview

FKAPI provides bulk endpoints optimized for fetching multiple kits in a single request. FootyCollect uses these endpoints to efficiently import user collections from the Football Kit Archive.

Bulk Fetch Kits

Method

Fetch multiple kits by their slugs in a single API call:

from footycollect.api.client import FKAPIClient

client = FKAPIClient()
slugs = ["arsenal-home-2023-24", "liverpool-away-2023-24", "barcelona-home-2023-24"]
kits = client.get_kits_bulk(slugs)

Implementation

Source: client.py:486-515

def get_kits_bulk(self, slugs: list[str]) -> list[dict]:
    """Get multiple kits by their slugs in a single request.
    
    Args:
        slugs: List of kit slugs (2-30 slugs supported)
    
    Returns:
        List of kit data with reduced response format:
        - name, team (name, logo, country), season (year), brand (name, logo), main_img_url
    """
    if not slugs:
        return []
    
    if len(slugs) < MIN_BULK_SLUGS:
        logger.warning("Bulk endpoint requires at least %d slugs, got %d", MIN_BULK_SLUGS, len(slugs))
        return []
    
    if len(slugs) > MAX_BULK_SLUGS:
        logger.warning("Bulk endpoint supports max %d slugs, got %d. Truncating.", MAX_BULK_SLUGS, len(slugs))
        slugs = slugs[:MAX_BULK_SLUGS]
    
    slugs_param = ",".join(slugs)
    logger.info("Fetching %d kits in bulk", len(slugs))
    result = self._get("/kits/bulk", params={"slugs": slugs_param})
    extracted = self._extract_list_from_result(result)
    logger.info("Bulk API extracted %d kits", len(extracted))
    return extracted

Constraints

int

default:"2"

Minimum number of slugs required for bulk endpointRequests with fewer than 2 slugs will return an empty list

int

default:"30"

Maximum number of slugs supported per requestRequests with more than 30 slugs will be automatically truncated

Defined in client.py:14-16:

# Bulk endpoint constraints
MIN_BULK_SLUGS = 2
MAX_BULK_SLUGS = 30

The bulk endpoint will truncate slug lists exceeding 30 items. For larger imports, split your requests into multiple batches of 30 slugs or fewer.

Response Format

Bulk responses use a reduced format for efficiency:

[
  {
    "name": "Arsenal Home 2023-24",
    "team": {
      "name": "Arsenal",
      "logo": "https://example.com/arsenal.png",
      "country": "England"
    },
    "season": {
      "year": "2023-24"
    },
    "brand": {
      "name": "Adidas",
      "logo": "https://example.com/adidas.png"
    },
    "main_img_url": "https://example.com/arsenal-home-2023.jpg"
  }
]

Bulk responses exclude some fields present in detailed kit responses to reduce payload size. Use get_kit_details() if you need complete kit information.

User Collection Import

Management Command

FootyCollect includes a Django management command for importing entire user collections from the Football Kit Archive:

python manage.py populate_user_collection <userid> [options]

Source: footycollect/collection/management/commands/populate_user_collection.py

Command Arguments

int

required

User ID from FootballKitArchiveExample: 12345

string

Username in your FootyCollect instance to assign items toDefault: Creates new user based on FKA profile

int

default:"120"

Maximum time to wait for scraping to complete (seconds)Default: 120 seconds

int

default:"20"

Number of items to fetch per pageDefault: 20

boolean

default:"false"

Dry run mode - preview what would be created without actually creating objectsUseful for testing imports

string

Path to JSON file with collection data (for testing/offline imports)Bypasses FKAPI calls

Usage Examples

# Import user 12345's collection
python manage.py populate_user_collection 12345

# Import to existing user 'johndoe'
python manage.py populate_user_collection 12345 --target-username=johndoe

# Preview what would be imported
python manage.py populate_user_collection 12345 --dry-run

# Wait up to 5 minutes for scraping
python manage.py populate_user_collection 12345 --wait-timeout=300

# Import from local JSON file
python manage.py populate_user_collection --json-file=/path/to/collection.json

How It Works

The populate_user_collection command performs the following steps: 1. Initiate Scraping Calls FKAPI to start scraping the user’s collection:

scrape_response = client.scrape_user_collection(userid)
# Returns: {"status": "queued", "task_id": "abc123"}

2. Wait for Completion Polls FKAPI until collection data is ready:

while time.time() - start_time < wait_timeout:
    response = client.get_user_collection(userid, page=1, page_size=20, use_cache=False)
    if response.get("status") in ("processing", "pending"):
        time.sleep(1)
        continue
    if response.get("data", {}).get("entries"):
        break

3. Fetch All Pages Iterates through paginated results:

all_entries = []
page = 1
while True:
    page_response = client.get_user_collection(userid, page=page, page_size=20)
    entries = page_response.get("data", {}).get("entries", [])
    if not entries:
        break
    all_entries.extend(entries)
    page += 1

4. Create Objects For each entry, creates:

Brand, Club, Season, TypeK, Competition, Kit (if they don’t exist)
BaseItem with metadata
Jersey with size and customization details
Photo objects from entry images

5. Download Assets Downloads club logos, brand logos, and kit photos automatically.

The command uses transactions to ensure atomicity - if an entry fails to import, it won’t leave partial data in the database.

FKAPI Client Methods

Scrape User Collection

def scrape_user_collection(self, userid: int) -> dict | None:
    """Start scraping user collection. Returns response with task_id or data."""
    from .tasks import scrape_user_collection_task
    
    scrape_user_collection_task.delay(userid)
    return {"status": "queued"}

Source: client.py:532-537

Get User Collection

def get_user_collection(
    self, userid: int, page: int = 1, page_size: int = 20, *, use_cache: bool = False
) -> dict | None:
    """Get user collection with pagination."""
    endpoint = f"/user-collection/{userid}"
    params = {"page": page, "page_size": page_size}
    return self._get(endpoint, params=params, use_cache=use_cache)

Source: client.py:539-546

int

default:"1"

Page number (1-indexed)

int

default:"20"

Number of entries per page

boolean

default:"false"

Whether to use cached resultsTypically disabled for user collection imports to ensure fresh data

Batch Processing Best Practices

Split Large Imports

For collections with hundreds of kits, split into batches:

def import_kits_in_batches(all_slugs: list[str]):
    client = FKAPIClient()
    batch_size = 30  # MAX_BULK_SLUGS
    
    for i in range(0, len(all_slugs), batch_size):
        batch = all_slugs[i:i + batch_size]
        
        # Skip if batch too small
        if len(batch) < 2:  # MIN_BULK_SLUGS
            continue
        
        kits = client.get_kits_bulk(batch)
        print(f"Fetched {len(kits)} kits from batch {i//batch_size + 1}")
        
        # Process kits...
        time.sleep(1)  # Rate limiting

Handle Failures Gracefully

The populate_user_collection command tracks success/failure:

created_count = 0
skipped_count = 0
error_count = 0

for entry in entries:
    try:
        if self._process_entry(entry, user, dry_run=dry_run):
            created_count += 1
        else:
            skipped_count += 1
    except Exception as e:
        error_count += 1
        logger.exception("Error processing entry")

print(f"Completed: {created_count} created, {skipped_count} skipped, {error_count} errors")

Avoid Duplicates

The command checks for existing items before creating:

def _find_existing_jersey_for_user_kit(self, user: User, kit: Kit | None) -> Jersey | None:
    """Return existing Jersey for user+kit if any."""
    if kit:
        return (
            Jersey.objects.filter(base_item__user=user, kit=kit)
            .select_related("base_item", "kit")
            .order_by("-base_item__created_at")
            .first()
        )
    return None

Source: populate_user_collection.py:722-736

Existing items are skipped to prevent duplicates. Re-running the command on the same user is safe.

Rate Limiting for Bulk Operations

Bulk operations are subject to the same rate limits as regular requests:

Client-side: 100 requests per minute
Each bulk request counts as 1 request, regardless of how many slugs it contains

Optimize Request Count

# ❌ Bad: 30 individual requests
for slug in slugs:
    kit = client.get_kit_details(slug)

# ✅ Good: 1 bulk request
kits = client.get_kits_bulk(slugs)

Error Handling

Empty Results

kits = client.get_kits_bulk(slugs)

if not kits:
    # Could mean:
    # - FKAPI unavailable
    # - No kits found for slugs
    # - Slugs list too short (< 2)
    logger.warning("No kits returned from bulk request")

Partial Failures

If some slugs are invalid, FKAPI returns only valid kits:

slugs = ["valid-slug-1", "invalid-slug", "valid-slug-2"]
kits = client.get_kits_bulk(slugs)

# Returns 2 kits (only valid ones)
print(f"Requested {len(slugs)} kits, got {len(kits)} kits")

Command Errors

The management command provides detailed error output:

$ python manage.py populate_user_collection 12345

Starting collection population for user ID: 12345
Starting scrape for user 12345...
Scrape response status: queued
Scrape request accepted, waiting for completion...
Collection ready
Fetching page 1...
  Fetched page 1 (20 entries, total: 20)
Fetching page 2...
No more entries on page 2, stopping pagination
Found 20 entries to process

[1/20] Processing entry 1 - Arsenal Home 2023-24...
  [OK] Entry 1 processed successfully
[2/20] Processing entry 2 - Liverpool Away 2023-24...
  ⚠ Entry 2 skipped (already exists)
...

Completed: 15 created, 5 skipped, 0 errors

Advanced Usage

Custom Pagination

Adjust page size for performance tuning:

# Smaller pages for faster initial response
response = client.get_user_collection(userid, page=1, page_size=10)

# Larger pages for fewer requests (max typically 50-100)
response = client.get_user_collection(userid, page=1, page_size=50)

Offline Import from JSON

For testing or offline scenarios:

# Export collection from FKAPI
curl http://fkapi/api/user-collection/12345 > collection.json

# Import without hitting FKAPI
python manage.py populate_user_collection --json-file=collection.json

Selective Import

Modify the command to filter entries:

# Only import jerseys from specific seasons
for entry in entries:
    season = entry.get('kit', {}).get('season', '')
    if season in ['2023-24', '2022-23']:
        self._process_entry(entry, user)

Performance Considerations

Database Transactions

Each entry is processed in a transaction for safety:

with transaction.atomic():
    brand = self._get_or_create_brand(...)
    club = self._get_or_create_club(...)
    # ... create all related objects
    
    if dry_run:
        transaction.set_rollback(True)  # Rollback in dry-run mode

Asset Downloads

Logos and photos are downloaded asynchronously to avoid blocking:

try:
    ensure_item_entity_logos_downloaded(base_item)
except Exception:
    logger.exception("Error downloading logos")
    # Continue processing - logos can be retried later

Large imports with many photos can take significant time. Use --dry-run first to estimate duration.

Constraints Summary

Constraint	Value	Behavior
MIN_BULK_SLUGS	2	Requests with < 2 slugs return empty list
MAX_BULK_SLUGS	30	Requests with > 30 slugs are truncated
Rate Limit	100/min	Requests exceeding limit return cached data
Request Timeout	60s	Requests exceeding timeout are retried
Max Retries	3	Failed requests retry with exponential backoff
Cache TTL	3600s	Cached responses expire after 1 hour

Next Steps

Review FKAPI overview for architecture and configuration
Learn about searching kits for individual kit queries
Explore the API reference for endpoint details

API Overview

Endpoints

FKAPI Integration

Bulk Operations with FKAPI

Overview

Bulk Fetch Kits

Method

Implementation

Constraints

Response Format

User Collection Import

Management Command

Command Arguments

Usage Examples

How It Works

FKAPI Client Methods

Scrape User Collection

Get User Collection

Batch Processing Best Practices

Split Large Imports

Handle Failures Gracefully

Avoid Duplicates

Rate Limiting for Bulk Operations

Optimize Request Count

Error Handling

Empty Results

Partial Failures

Command Errors

Advanced Usage

Offline Import from JSON

Selective Import

Performance Considerations

Database Transactions

Asset Downloads

Constraints Summary

Next Steps

​Overview

​Bulk Fetch Kits

​Method

​Implementation

​Constraints

​Response Format

​User Collection Import

​Management Command

​Command Arguments

​Usage Examples

​How It Works

​FKAPI Client Methods

​Scrape User Collection

​Get User Collection

​Batch Processing Best Practices

​Split Large Imports

​Handle Failures Gracefully

​Avoid Duplicates

​Rate Limiting for Bulk Operations

​Optimize Request Count

​Error Handling

​Empty Results

​Partial Failures

​Command Errors

​Advanced Usage

​Custom Pagination

​Offline Import from JSON

​Selective Import

​Performance Considerations

​Database Transactions

​Asset Downloads

​Constraints Summary

​Next Steps

Overview

Bulk Fetch Kits

Method

Implementation

Constraints

Response Format

User Collection Import

Management Command

Command Arguments

Usage Examples

How It Works

FKAPI Client Methods

Scrape User Collection

Get User Collection

Batch Processing Best Practices

Split Large Imports

Handle Failures Gracefully

Avoid Duplicates

Rate Limiting for Bulk Operations

Optimize Request Count

Error Handling

Empty Results

Partial Failures

Command Errors

Advanced Usage

Custom Pagination

Offline Import from JSON

Selective Import

Performance Considerations

Database Transactions

Asset Downloads

Constraints Summary

Next Steps