perf + feat(steam-sniper): catalog 1.5-2.3x faster + Steam Market delta indicator#53
Open
perf + feat(steam-sniper): catalog 1.5-2.3x faster + Steam Market delta indicator#53
Conversation
…pre-cached cat/model User asked if anything could make the site faster. Profiled prod, found: - /api/catalog?q=karambit: 794ms - /api/catalog?limit=200: 881ms - /api/catalog?limit=50: 666ms Root: get_catalog() was building enriched dicts for ALL ~23k items first (classify + _weapon_model + _get_item_image + _calc_trend), then filtering down to 50-200 for the page. 99% of the work was thrown away. Two optimizations: 1. Filter-before-enrich in get_catalog(). Stage 1 walks _prices with O(1) checks (category/state/q/model/sort), collects (raw_item, cat, model) tuples. Stage 2 enriches only the final page (50-200 items). Image lookup, trend calc, dict construction now run on page only. 2. Pre-compute _cat / _model fields on each item dict in _collect_once (5-min refresh). get_catalog reads item["_cat"] / item["_model"] instead of calling classify() / _weapon_model() per request. /api/lists also uses item_data["_cat"] for category. Tests fall back to live classify() if cache field absent (TestClient skips collector). Results (prod, median of 4 runs): /api/catalog?limit=50&offset=0 666ms → 409ms (1.63x) /api/catalog?limit=50&category=case 460ms → 350ms (1.32x) /api/catalog?limit=50&q=karambit 794ms → 343ms (2.32x) /api/catalog?limit=200&offset=0 881ms → 571ms (1.54x) Network round-trip floor is ~220ms (RU→msk-1 + uvicorn). Server-side work on catalog dropped from ~440ms to ~190ms for limit=50 — about 2.3x on the actual computation. /api/lists barely moved (~457→429ms) — its bottleneck is _get_item_image (regex + multi-lookup) per item. Left alone for now (diminishing returns, 50-100ms not worth a deeper refactor). Tests: 133/133 passed. No behavioral changes — only ordering of operations and field caching. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…kins
Adds a column showing the difference between lis-skins price and Steam
Market median price for each item. Helps spot real arbitrage (lis-skins
20-30% below Steam after Steam's 15% commission floor) vs paying premium
for instant delivery.
Spike confirmed: 10/10 items from lesha's favorites are 25-48% below
Steam Market. The delta is meaningful information.
## Backend
- `db.steam_prices` table: {name, lowest_usd, median_usd, volume, fetched_at}
Migrated automatically by db.init_db().
- `db.upsert_steam_price` / `db.get_steam_prices` / `db.get_steam_price` helpers.
- `_steam_priceoverview_sync()` calls Steam's priceoverview API
(https://steamcommunity.com/market/priceoverview/) per item. ~400ms each.
- `_refresh_steam_prices()` background pass: walks user_lists ∪ watchlist names,
rate-limited to 40 req/min (1.5s between calls). ~3-5 min per full pass on
~200 items. Result cached in `_steam_prices_cache` (also persisted in DB so
cache survives restart).
- `_steam_prices_loop()` re-runs every STEAM_REFRESH_INTERVAL (1h).
- `_compute_steam_delta(name, lis_rub)` computes {steam_price_rub, delta_pct,
volume, fetched_at}. Uses median_usd (more stable than lowest on low-volume
items). Removes lis-skins markup (LIS_SKINS_RATE_MULTIPLIER) for fair compare.
- Enrichment in /api/catalog (page only — fits filter-before-enrich pattern),
/api/lists, /api/item/{name}.
## Frontend
- `utils.js` `steamDeltaBadge(steam)` — compact badge next to price:
delta < -15%: green (real deal — beats Steam's 15% commission)
delta -15..+5%: gray (parity)
delta > +5%: orange (paying premium)
⚠ marker if Steam volume < 5/week (price unreliable).
- catalog.js, lists.js, cases.js — render badge on each card.
- item_detail.js — full block "Steam median: X ₽ • Δ ±N% • vol N/нед" on
detail page. Replaces previously broken sign logic in renderPriceCompare.
- styles.css — colored chip variants for both compact (.steam-delta) and
full (.detail-price-chip) presentations.
## Why median, not lowest
Lowest_price on Steam can be a single overpriced listing. Median reflects
actual transaction history over the past week. For low-volume items
(<5 sales/week), even median is shaky — UI shows ⚠.
## What's NOT addressed
- Items not in user lists/watchlist won't show delta on catalog browse —
Steam rate limit makes refreshing all 23k infeasible (would take ~14 hours).
Only items someone cares about get tracked. Acceptable for personal tool.
- Steam currency=1 (USD). Conversion uses _lis_rate() and reverses the
lis-skins markup. Not perfect (RUB→USD round-trips), but good enough
for an indicator.
Tests: 133/133 passed. No behavior change to existing endpoints when
Steam cache empty (steam field simply absent from response).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
NickStr11
changed the title
Earlier deploy_quick.py listed a hand-picked subset of JS modules and
silently dropped the rest (utils.js, alerts.js, chart.js, events.js,
modal.js, router.js, search.js, state.js, stats.js, theme.js).
Adding a new export to utils.js (steamDeltaBadge) and importing it from
catalog.js shipped a broken module: catalog.js with new import landed on
prod, utils.js stayed stale → JS module load failed → entire dashboard
went blank ("ничего нажать не могу"). Hot-fixed by uploading utils.js
manually and restarting the service.
Now FILES includes every js file in static/js/. Safer than selective list.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…gest
Three features in one branch — all сшиты на lis-skins, без multi-marketplace.
## 1. Float-aware target alerts
Alert "below" can now require BOTH price ≤ target_below AND a listing with
float in [target_float_min, target_float_max] to fire. Without float bounds,
alert previously triggered on listings with the right price but unusable
wear (e.g. 0.78 float on a knife the user wanted ≤0.10).
- db.user_lists: + target_float_max, target_float_min (auto-migrated)
- db.set_list_item_targets: + float kwargs
- _check_list_alerts: when float bounds set, query listings_snapshot.db for
cheapest listing matching float range; trigger only if its rub ≤ target_below
- PATCH /api/lists/target: + float fields with [0,1] range validation
- Item-detail UI: + Float ≤ / Float ≥ inputs with [0,1] step=0.001
- Float bounds apply only to BELOW direction (buying) — not relevant for ABOVE
## 2. Inline highlight on lis-skins catalog pages
Extension v1.6 scans cards on every lis-skins page, asks backend which
items are in user's lists, paints a colored ring around them:
- ♥ red ring → in favorite
- ★ yellow ring → in wishlist
- ♥★ orange ring → in both (Sniper brand color)
Tooltip shows targets on hover. Re-scans on DOM mutations (lazy-loaded cards).
- New endpoint GET /api/lists/check?user=&urls=...
Resolves lis-skins URLs via slug index, returns membership + targets per URL.
Accepts pipe-separated urls or names. Items not in lists omitted.
- extension/content.js: findCardsOnPage + scanAndHighlight + scheduleScan
on MutationObserver. Skips category-root URLs.
- extension/background.js: + checkLists() with 8s timeout
- extension/styles.css: .sniper-highlight* classes with corner badges
- extension/manifest.json: 1.5 → 1.6
## 3. Weekly TG digest (Sunday 18:00 MSK)
Standalone digest.py module, scheduled via systemd timer. Composes:
- 🔥 Top price drops in lists (week-over-week)
- 🎯 Alerts that fired during the week
- 💎 Best Steam Market arbitrages (Δ < -15%, vol ≥ 5/нед)
- 📉 Stale items (added >30 days ago, no alerts triggered)
Sends nothing if all sections empty (avoids weekly "no data" spam).
- new digest.py — pure Python, imports only db; no FastAPI dependency
- new deploy/steam-sniper-digest.{service,timer}
- _collect_once now snapshots prices for user_lists too (not just watchlist) —
required for week-over-week diff in digest
- deploy.py: + new systemd units in SYSTEMD_UNITS
- deploy_quick.py: + digest.py to FILES
## Production verification
- 133/133 tests passed (pre-commit hook)
- Quick deploy on http://72.56.37.150/ — dashboard active
- /api/lists/check tested with smoke
- digest.timer enabled, next run Sun 2026-05-10 18:00 MSK
- digest --dry-run on prod returned "no data to report" cleanly (snapshots
for list-items just started accumulating; will populate over the week)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Содержит две связанные правки на той же ветке:
1. perf: catalog 1.5-2.3× faster (commit fcc7ee6)
Filter-before-enrich + pre-cached
_cat/_modelв_collect_once./api/catalog?q=karambit/api/catalog?limit=50/api/catalog?limit=200/api/catalog?category=case2. feat: Steam Market delta indicator (commit 2cbeaab)
Колонка с разницей цен lis-skins vs Steam Market median. Помогает спотить реальный арбитраж.
Backend:
steam_pricesв SQLite + helpers_steam_priceoverview_sync()— Steam priceoverview API_refresh_steam_prices()— rate-limited bg task на watched items (1.5s/req)Frontend:
steamDeltaBadge()chip рядом с ценой:Steam median: X ₽ • Δ ±N% • vol N/недProduction verification (90s после deploy):
Все items в зелёной зоне -20…-40% → реальный арбитраж.
Trade-offs:
_lis_rate()с откатом lis-skins markup — не идеал, но достаточно для индикатора.Test plan
pytest tests/— 133/133 passed🤖 Generated with Claude Code