53 Commits

Author	SHA1	Message	Date
Egor Aristov	8f12325b7c	adblock	2025-03-22 11:53:13 +03:00
Egor Aristov	a9ee50d722	print config	2025-03-21 11:31:30 +03:00
Egor Aristov	1e42eb81ea	retries	2025-03-20 22:43:36 +03:00
Egor Aristov	a0876409c5	retries	2025-03-20 22:27:36 +03:00
Egor Aristov	8da176ffba	small upd	2025-03-19 14:35:46 +03:00
Egor Aristov	87f7c017de	increase screenshot height	2025-03-18 14:04:35 +03:00
Egor Aristov	45bb528917	removed waiting for 1st post (on some sites "1 post is fully rendered" != "whole page loaded")	2025-03-18 14:04:32 +03:00
Egor Aristov	6e5c657a18	some useful improvements for debugging	2025-03-18 14:04:31 +03:00
Egor Aristov	94b347e49f	use dummy limiter for testing pages	2025-03-18 14:04:29 +03:00
Egor Aristov	67c6bd9856	protobuf	2025-03-18 14:04:14 +03:00
Egor Aristov	b9f77388a1	unit tests...	2025-03-18 14:03:52 +03:00
Egor Aristov	220bc50e47	block local ips and service worker; util function to resolve arbitrary 'host-like' string, unit testing	2025-03-18 14:03:52 +03:00
Egor Aristov	08326debdd	block local ips and service worker; util function to resolve arbitrary 'host-like' string, unit testing	2025-03-18 14:03:52 +03:00
Egor Aristov	8d21583f9e	prevent webrtc leak through proxy	2025-03-18 14:03:52 +03:00
Egor Aristov	83ffba44f7	log visit time to "info" instead of "debug"	2025-03-18 14:03:51 +03:00
Egor Aristov	bca61c5bd3	opt-in to "new headless"	2025-03-18 14:03:51 +03:00
Egor Aristov	5aba78ed66	by domain rate limiter	2025-03-18 14:03:51 +03:00
Egor Aristov	4653c691c6	better ip detection	2025-03-18 14:03:51 +03:00
Egor Aristov	86108b4c90	better ip detection	2025-03-18 14:03:50 +03:00
Egor Aristov	d654ce7c8b	rate limits	2025-03-18 14:03:50 +03:00
Egor Aristov	caaf410a70	yaml configuration removed to simplify deploy and development	2025-03-18 14:03:50 +03:00
Egor Aristov	72ec41dccf	extractor rewrite (use custom wrappers over playwright locator)	2025-03-18 14:03:50 +03:00
Egor Aristov	aab8c026f4	some refactoring (sol_I_d)	2025-03-18 14:03:50 +03:00
Egor Aristov	8203955494	stream name parameter	2025-03-18 14:03:49 +03:00
Egor Aristov	82f3dc5c21	check version in backend	2025-03-18 14:03:48 +03:00
Egor Aristov	69cf5ba95a	check version in backend	2025-03-18 14:03:48 +03:00
Egor Aristov	540a4a6a00	embed fs	2025-03-18 14:03:47 +03:00
Egor Aristov	0d7c238e65	fix: skip posts with empty date	2025-03-18 14:02:36 +03:00
Egor Aristov	f38f083d63	entry ids now use url with query	2025-03-18 14:02:36 +03:00
Egor Aristov	da66b8de73	log cookie header	2025-03-18 14:02:36 +03:00
Egor Aristov	f956d883bf	change timeouts	2025-03-18 14:02:36 +03:00
Egor Aristov	6c51d7b24b	cookie manager	2025-03-18 14:02:35 +03:00
Egor Aristov	31482d274c	dont discard already accepted task	2025-03-18 14:02:35 +03:00
Egor Aristov	09701e68f4	fix null pointer	2025-03-18 14:02:35 +03:00
Egor Aristov	dc7c3482c9	increase task timeout	2025-03-18 14:02:35 +03:00
Egor Aristov	925bf49a8e	compare revs (for testing) ... Support mocking dates Support mocking dates Support mocking dates Support mocking dates	2025-03-18 14:02:34 +03:00
Egor Aristov	0d48fe8554	date parsing enhancements	2025-03-18 14:01:43 +03:00
Egor Aristov	8e7e1d1cd6	terminate task in case of panic	2025-03-18 14:01:42 +03:00
Egor Aristov	f78345ab14	allow empty description or content	2025-03-18 14:01:42 +03:00
Egor Aristov	c46706c32e	fix: fill headers for render task	2025-03-18 14:01:42 +03:00
Egor Aristov	ef7433a594	cookies fix	2025-03-18 14:01:41 +03:00
Egor Aristov	fed51635f5	headers and cookies support	2025-03-18 14:01:41 +03:00
Egor Aristov	bd33314298	screenshots; some refactoring	2025-03-18 14:01:41 +03:00
Egor Aristov	ca2087eb7b	move http handler to separate package	2025-03-18 14:01:41 +03:00
Egor Aristov	e3ad088dbd	change consumer to durable	2025-03-18 14:01:41 +03:00
Egor Aristov	155cb37735	speed up loading by waiting for EITHER locator OR networkIdle	2025-03-18 14:01:40 +03:00
Egor Aristov	87ceeb4376	prevent resubmitting already running task	2025-03-18 14:01:40 +03:00
Egor Aristov	031b062277	remove nats-msg-id	2025-03-18 14:01:40 +03:00
Egor Aristov	0c5ad7e692	debugging nats...	2025-03-18 14:01:40 +03:00
Egor Aristov	94694b2fee	proxy support; config validation	2025-03-18 14:01:40 +03:00