RL: encoding observacii, akcne masky a Round prostredie
Zaklad RL vrstvy (rl/DESIGN.md): egocentricky rotovana observacia (ruka, videne karty, tipy, kopka, dedukovane voidy -- 233 dim), masky legalnych tipov/kariet zrkadliace pravidla enginu a RoundEnv (jedno kolo = jedna self-play epizoda). Fuzz-testy vynucuju zhodu masiek s enginom. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Claude Fable 5
+181
@@ -0,0 +1,181 @@
|
||||
# RL bot pre bridzik — navrh (2026-07-01)
|
||||
|
||||
Ciel: naucit sa principy self-play reinforcement learningu (v duchu AlphaGo Zero)
|
||||
na praktickom priklade — natrenovat sietovy policy pre hranie bridziku. Zamerne
|
||||
zjednodusene oproti AlphaGo Zero: bez MCTS (skryta informacia neumoznuje priamy
|
||||
prehladavaci strom), cisty self-play policy gradient (PPO/REINFORCE) nad `bridzik.py`
|
||||
enginom.
|
||||
|
||||
## Kluc: `Round` je nezavisla epizoda
|
||||
|
||||
Bodovanie (`Round.get_points_summary`) je cisto lokalne pre jedno kolo — nezavisi
|
||||
od predoslych ani nasledujucich kol, len od tipu a poctu kopiek v danom kole.
|
||||
Netreba teda simulovat cely `Bridzik`/`Series` state machine na trening — staci
|
||||
instanciovat `Round(round_number, first_player, shuffler)` priamo, opakovane,
|
||||
s roznymi `round_number` (0-7, teda 8 az 1 karta v ruke). Kazdy `Round` je
|
||||
samostatna self-play epizoda.
|
||||
|
||||
Vyhody:
|
||||
- jednoduchsi self-play loop (ziadne series/game bookkeeping)
|
||||
- vela nezavislych epizod, jednoduchá paralelizacia
|
||||
|
||||
**Curriculum vs. uniformne samplovanie (bod 4).** Povodny napad "zacat na
|
||||
`round_number=7`" je zavadzajuci: pri 1 karte je tip masked na {0,1} a jedina
|
||||
karta je vynutena — nula card-play rozhodnuti, ziadne ucenie, len smoke-test.
|
||||
Realne ucenie je v kolach `round_number` 0-3 (6-8 kariet). Preto:
|
||||
- `round_number` je aj tak v observacii, takze **default = uniformne samplovat
|
||||
`round_number` 0-7** a nechat siet zdielat vahy naprieč velkostami ruk.
|
||||
- Ak curriculum, tak **od tazkych (viac kariet) k lahsim**, nie naopak; alebo
|
||||
aspon uniformne s miernym zvyhodnenim tazsich kol.
|
||||
- `round_number=7` drzat len ako sanity/smoke test pipeline, nie ako trening.
|
||||
|
||||
## 1. State encoding (observation)
|
||||
|
||||
Spolocne pre guess aj play fazu, budovane z `Round` objektu pre daneho hraca:
|
||||
|
||||
- **vlastna ruka** — 32-dim multi-hot (4 farby x 8 hodnot)
|
||||
- **round_number** — one-hot (8) alebo normalizovane cislo (urcuje velkost ruk)
|
||||
- **tipy vsetkych 4 hracov** — 4x (flag "uz tipoval" + normalizovana hodnota)
|
||||
- **vlastny tip** (po tipnuti) — kriticke pre play fazu (viem, ci este potrebujem
|
||||
vyhrat kopku, alebo sa jej mam vyhybat)
|
||||
- **kolko kopiek uz kazdy hrac vyhral v tomto kole** — 4 scalars, odvodene
|
||||
z dokoncenych `Stash` objektov v `self.stashes`
|
||||
- **aktualna kopka v procese** — 4 sloty (karta alebo prazdne) + `first_player`
|
||||
aktualnej kopky
|
||||
- **uz odohrane/videne karty v tomto kole** — 32-dim multi-hot (bod 1). KRITICKE:
|
||||
bez toho observacia NIE JE Markovovska. Pri viac kartach (round_number 0-2)
|
||||
su dve rovnake ruky s rovnakou aktualnou kopkou, ale roznou historiou uz
|
||||
odohranych kariet, rozne stavy s roznym optimalnym tahom (vies, ci este visi
|
||||
eso/cerven). Bez tejto zlozky sa siet nemoze naucit card-counting a strop hry
|
||||
ostane nizky. Kodovat karty odohrane v predoslych dokoncenych `Stash`-och
|
||||
(mimo tvojej ruky a mimo aktualnej rozohranej kopky).
|
||||
- **(volitelne, neskor — bod 8) znama neúčasť supperov vo farbe (voids)** —
|
||||
ked supper neprizna vynasanu farbu, prezradi void → per-hrac x per-farba
|
||||
flag. Silna informacia, ale nechat na neskorsie rozsirenie.
|
||||
|
||||
**Egocentricka rotacia (bod 2) — povinny invariant.** Aby parameter sharing
|
||||
medzi 4 sedadlami fungoval, VSETKY 4-hracske vektory (tipy, pocty vyhranych
|
||||
kopiek, sloty aktualnej kopky, `first_player`) musia byt rotovane tak, ze
|
||||
"ja" = index 0 a ostatni relativne (+1, +2, +3 v smere hry). Toto zapisat do
|
||||
`encoding.py` ako tvrdy invariant + unit test — je to najpravdepodobnejsie
|
||||
miesto tichej chyby, ktora pokazi ucenie.
|
||||
|
||||
Zamerne vynechane: priebezne skore/standings naprieč hrou — kedze odmena je
|
||||
per-round nezavisla, optimalne rozhodnutie v danom kole na standings nezavisi.
|
||||
|
||||
**Velkost observacie (bod 5).** Povodny odhad ~80-100 floatov je podstrelený.
|
||||
Ak sa 4 sloty aktualnej kopky koduju one-hot (4x32=128) + ruka 32 + videne
|
||||
karty 32 + tipy/pocty/round_number/first_player, realny `input_dim` je skor
|
||||
~200. Nie je to problem, len podla toho nastavit vstupnu vrstvu siete.
|
||||
|
||||
## 2. Akcny priestor + maskovanie
|
||||
|
||||
- **Guess**: 9 kategorii (0-8), maskovane na `0..(8-round_number)`; pre 4.
|
||||
(posledneho) tipujuceho naviac zamaskovat hodnotu, ktora by sposobila
|
||||
`BridzikException` (sucet tipov = pocet kopiek) — vypocitatelne vopred
|
||||
z `self.guesses`. Zakazana hodnota = `(8-round_number) - sum(3 tipov)`;
|
||||
ak vyjde mimo `0..(8-round_number)`, je uz aj tak nelegalna a nemaskuje sa
|
||||
nic navyse (osetrit rozsah).
|
||||
- **Play card**: 32 kategorii (rovnaka indexacia farba+hodnota ako hand-encoding),
|
||||
maskovane na karty, ktore hrac realne ma A splnaju follow-suit pravidlo
|
||||
(rovnaka logika ako v `Round.play_card`: farba prvej karty v kopke, inak
|
||||
povinna cervena ak ju hrac ma).
|
||||
|
||||
## 3. Sieť
|
||||
|
||||
Zdielany "trup" (2 hidden layers, ~128-256 neuronov, ReLU) nad observation
|
||||
vektorom (~80-100 floatov), s troma vystupmi:
|
||||
|
||||
- guess head (9 logitov)
|
||||
- play head (32 logitov)
|
||||
- value head (1 scalar) — odhad ocakavanej odmeny do konca kola (baseline
|
||||
pre actor-critic)
|
||||
|
||||
`input_dim` nastavit podla realnej velkosti observacie (~200, viz bod 5
|
||||
v sekcii 1), nie podla povodneho ~80-100.
|
||||
|
||||
Fazovy flag v observacii + maskovanie urcuje, ktora hlava je pouzitelna
|
||||
v danom kroku (guess a play fazy sa nikdy neprelinaju).
|
||||
|
||||
## 4. Odmena a trening
|
||||
|
||||
- Odmena = 0 pocas kola; na konci kola kazdy hrac dostane
|
||||
`points_summary[player]` (0 alebo `10+guess`) ako terminalnu odmenu za
|
||||
VSETKY svoje rozhodnutia v danom kole (guess + vsetky `play_card` tahy).
|
||||
Sparse terminal reward, ziadne discountovanie netreba — `gamma=1` (bod 9),
|
||||
kolo ma max 9 rozhodnuti na hraca: round 0 = 1 tip + 8 kariet.
|
||||
- Algoritmus: self-play PPO (prip. najprv jednoduchsie REINFORCE + baseline),
|
||||
jedna zdielana siet hra vsetkych 4 hracov v kazdom `Round` (parameter
|
||||
sharing, rovnaky princip ako AlphaGo Zero).
|
||||
- **Normalizacia odmeny (bod 7).** `10+guess` je v rozsahu 10-18 a lisi sa
|
||||
per kolo; pri miesanych `round_number` to zvysuje varianciu policy gradientu.
|
||||
Standardizovat advantage per batch (odcitat priemer, delit std) — bezna
|
||||
PPO praktika, tu je nutnejsia kvoli rozne velkym odmenam.
|
||||
- Paralelizacia: `Round` instancie su nezavisle bez shared state, self-play
|
||||
generovanie sa da paralelizovat cez multiprocessing naprieč jadrami CPU
|
||||
(pripadne batchovanim viacerych epizod naraz cez sietovy forward).
|
||||
|
||||
**Caveat: hra nie je zero-sum (bod 3).** Je to 4-hracska general-sum hra —
|
||||
viacero hracov moze naraz trafit tip a vsetci skoruju, zaroven sa o kopky
|
||||
sutazi (`sum(kopky) = pocet kopiek`). Self-play so zdielanymi vahami preto
|
||||
NEMA konvergencne zaruky ako AlphaGo Zero (2-hracska zero-sum); skonverguje
|
||||
k *nejakemu* equilibriu, nie nutne k optimu, a moze oscilovat. Na ucebny
|
||||
ciel to staci, ale: (a) nepredavat si to ako "AlphaZero-grade optimalitu",
|
||||
(b) sledovat progres proti FIXNYM baseline-om (sekcia 5), nie len podla
|
||||
self-play reward, ktory sa hybe s protihracom.
|
||||
|
||||
## 5. Vyhodnotenie
|
||||
|
||||
- priemerne body/kolo oproti baseline (nahodny legalny hrac, Monte Carlo
|
||||
heuristicky tipper — pozri sekciu nizsie)
|
||||
- presnost tipu (% kôl, kde sa tip presne trafil) — interpretovatelnejsia
|
||||
metrika nez surove body
|
||||
|
||||
## Alternativa/doplnok pre tipovaciu fazu: Monte Carlo namiesto siete
|
||||
|
||||
Kedze tipovanie je v podstate odhad pravdepodobnosti pri neznamom rozdeleni
|
||||
zvysnych kariet, da sa riesit aj bez siete:
|
||||
|
||||
1. **Naivna MC simulacia** — vygenerovat vela nahodnych rozdeleni zvysnych
|
||||
kariet medzi ostatnych 3 hracov, odsimulovat kolo s jednoduchou heuristickou
|
||||
hracou strategiou, spocitat rozdelenie poctu vlastnych kopiek. POZOR: kedze
|
||||
bodujeme len presnu zhodu, spravny cieľ je **mod** rozdelenia, nie priemer.
|
||||
POZOR 2 (bod 6 — "discard pile"): `deal_starting_cards` zahodí prvych
|
||||
`4*round_number` kariet (`round_cards[4*round_number:]`), takze v kole NIE
|
||||
su rozdane vsetky karty. MC teda z `32 - vlastna_ruka` kariet rozdá kazdemu
|
||||
z 3 supperov len `(8-round_number)` kariet a **zvysok necha v neznamej kope
|
||||
mimo hru** — nerozdavat vsetko medzi supperov, inak nadhodnotis, kolko
|
||||
vysokych kariet/cervene supperi drzia.
|
||||
2. **Silnejsia verzia** — rovnaky MC rollout, ale simulovat zvysok kola
|
||||
s uz natrenovanou card-play sietou namiesto naivnej heuristiky (analogia
|
||||
MCTS + value network v AlphaZero namiesto ciste nahodnych rolloutov).
|
||||
|
||||
Toto sa da pouzit ako rychly heuristicky baseline bez trenovania siete na
|
||||
tipovanie vobec, alebo ako silnejsi hybrid s uz existujucou play sietou.
|
||||
|
||||
## Poradie implementacie
|
||||
|
||||
1. `rl/encoding.py` — cistě funkcie observation + mask (nad `Round` objektom),
|
||||
testovatelne izolovane. Uz tu zapracovat bod 1 (videne karty) a bod 2
|
||||
(egocentricka rotacia). Unit/property testy: maska NIKDY nepovoli tah, ktory
|
||||
`Round.play_card`/`add_player_guess` odmietne (fuzz-test proti enginu);
|
||||
rotacia je konzistentna pre vsetky 4 sedadla.
|
||||
2. `rl/env.py` — step/reset wrapper okolo jedneho `Round` (nie celeho `Bridzik`)
|
||||
3. **baseline hraci + evaluacny harness UZ TU** (nahodny legalny hrac, MC
|
||||
heuristicky tipper podla sekcie vyssie) — nech je metrika k dispozicii od
|
||||
prvej trenovacej epochy a da sa sledovat progres (bod 3: proti fixnym
|
||||
baseline-om, nie len self-play reward).
|
||||
4. sieť (PyTorch, trup + 3 hlavy) — `input_dim` podla realnej velkosti obs (~200)
|
||||
5. self-play generator (paralelne `Round` epizody)
|
||||
6. PPO update krok (advantage standardizovat per batch — bod 7)
|
||||
7. (neskor, volitelne) rozsirit na cely `Series`/`Bridzik` self-play, ak by
|
||||
sa ukazalo, ze cross-round dynamika (rotacia first_player a pod.) predsa
|
||||
len nieco mení — podla bodu 1 to nie je ocakavane
|
||||
|
||||
## Technologie
|
||||
|
||||
- **PyTorch** — samostatny `requirements-rl.txt`, oddeleny od zakladneho
|
||||
`requirements.txt` projektu
|
||||
- vlastna mensia implementacia PPO (v duchu CleanRL) namiesto Stable-Baselines3/
|
||||
RLlib — nas pripad (multi-agent self-play so zdielanymi vahami, maskovanie
|
||||
akcii) sa bije s ich single-agent Gym abstrakciou viac, nez by pomohla
|
||||
+168
@@ -0,0 +1,168 @@
|
||||
"""Observation a action-mask encoding nad `Round` objektom z `bridzik.py`.
|
||||
|
||||
Ciste funkcie bez externych zavislosti (ziadny numpy/torch) -- vystupy su
|
||||
obycajne zoznamy floatov/boolov, konverzia na tenzory je vecou volajuceho.
|
||||
|
||||
Tvrdy invariant (viz rl/DESIGN.md): VSETKY 4-hracske zlozky observacie su
|
||||
egocentricky rotovane -- "ja" (parameter `player`) je vzdy index 0, ostatni
|
||||
hraci +1/+2/+3 v smere hry. Bez tejto rotacie parameter sharing medzi
|
||||
sedadlami nefunguje.
|
||||
|
||||
Masky zrkadlia pravidla enginu: maska NIKDY nesmie povolit akciu, ktoru by
|
||||
`Round.add_player_guess` / `Round.play_card` odmietli, a naopak. Tuto zhodu
|
||||
vynucuje fuzz-test v tests/test_encoding.py.
|
||||
"""
|
||||
|
||||
from bridzik import Card, Card_colors, Card_values, ROUNDS_PER_SERIES
|
||||
|
||||
# Kanonicke poradie farieb a hodnot = poradie deklaracie v enum-och.
|
||||
COLORS = list(Card_colors) # HEARTS, LEAVES, ACORNS, BELLS
|
||||
VALUES = list(Card_values) # C7, C8, C9, C10, LOWER, UPPER, KING, ACE
|
||||
|
||||
N_CARDS = 32
|
||||
N_GUESS_ACTIONS = 9 # tipy 0..8
|
||||
N_PLAY_ACTIONS = N_CARDS # jedna akcia = jedna karta
|
||||
|
||||
_COLOR_INDEX = {color: i for i, color in enumerate(COLORS)}
|
||||
_VALUE_INDEX = {value: i for i, value in enumerate(VALUES)}
|
||||
|
||||
# Layout observacie -- offsety su sucast verejneho kontraktu (testy aj siet
|
||||
# sa na ne odkazuju menom, nie magickym cislom).
|
||||
OFF_HAND = 0 # 32 multi-hot: vlastna ruka
|
||||
OFF_SEEN = OFF_HAND + N_CARDS # 32 multi-hot: karty z dokoncenych kopiek
|
||||
OFF_ROUND = OFF_SEEN + N_CARDS # 8 one-hot: round_number
|
||||
OFF_PHASE = OFF_ROUND + ROUNDS_PER_SERIES # 1 flag: 1.0 = tipovacia faza
|
||||
OFF_GUESSES = OFF_PHASE + 1 # 4 x (flag "uz tipoval", tip/8), rel. poradie
|
||||
OFF_TRICKS = OFF_GUESSES + 8 # 4 x (vyhrane kopky / 8), rel. poradie
|
||||
OFF_STASH = OFF_TRICKS + 4 # 4 x 32 one-hot: aktualna kopka, rel. poradie
|
||||
OFF_STASH_LEADER = OFF_STASH + 4 * N_CARDS # 4 one-hot: rel. first_player kopky
|
||||
OFF_VOIDS = OFF_STASH_LEADER + 4 # 4 hraci x 4 farby: dedukovane voidy, rel. poradie
|
||||
OBS_DIM = OFF_VOIDS + 16 # = 233
|
||||
|
||||
|
||||
def card_index(card: Card) -> int:
|
||||
"""Index karty 0..31: farba (blok po 8) + hodnota."""
|
||||
return _COLOR_INDEX[card.color] * 8 + _VALUE_INDEX[card.value]
|
||||
|
||||
|
||||
def index_card(index: int) -> Card:
|
||||
"""Inverzia card_index."""
|
||||
return Card(COLORS[index // 8], VALUES[index % 8])
|
||||
|
||||
|
||||
def relative_seat(seat: int, player: int) -> int:
|
||||
"""Egocentricka rotacia: `player` -> 0, dalsi v smere hry -> 1, 2, 3."""
|
||||
return (seat - player) % 4
|
||||
|
||||
|
||||
def deduce_voids(rnd) -> dict:
|
||||
"""Isto-dedukovane chybajuce farby hracov z priebehu kola.
|
||||
|
||||
Follow-suit pravidlo prezradza: kto nepriznal vynasanu farbu, uz ju nema;
|
||||
kto pri tom nezahral ani cerven (tromf), nema ani tu. Vynasajuci hrac
|
||||
neprezradza nic. Vracia dict seat -> set(Card_colors). Kedze karty pocas
|
||||
kola len ubudaju, raz dedukovany void plati do konca kola.
|
||||
"""
|
||||
voids = {seat: set() for seat in range(4)}
|
||||
for stash in rnd.stashes:
|
||||
first = stash.get_first_card()
|
||||
if first is None:
|
||||
continue
|
||||
for seat, card in stash.get_cards().items():
|
||||
if seat == stash.first_player:
|
||||
continue
|
||||
if card.color != first.color:
|
||||
voids[seat].add(first.color)
|
||||
if card.color != Card_colors['HEARTS']:
|
||||
voids[seat].add(Card_colors['HEARTS'])
|
||||
return voids
|
||||
|
||||
|
||||
def encode_observation(rnd, player: int) -> list:
|
||||
"""Observacia kola z pohladu hraca `player` (OBS_DIM floatov v [0, 1]).
|
||||
|
||||
Funguje v oboch fazach (tipovanie aj hra) aj na terminalnom stave;
|
||||
necita get_active_player(), takze sa da zavolat pre lubovolneho hraca
|
||||
kedykolvek.
|
||||
"""
|
||||
obs = [0.0] * OBS_DIM
|
||||
|
||||
for card in rnd.player_cards[player]:
|
||||
obs[OFF_HAND + card_index(card)] = 1.0
|
||||
|
||||
# Dokoncene kopky -> "videne karty"; jedina pripadna nedokoncena kopka
|
||||
# (posledna) je aktualna rozohrana a koduje sa do slotov nizsie.
|
||||
current_stash = None
|
||||
for stash in rnd.stashes:
|
||||
if stash.is_completed():
|
||||
for card in stash.get_cards().values():
|
||||
obs[OFF_SEEN + card_index(card)] = 1.0
|
||||
else:
|
||||
current_stash = stash
|
||||
|
||||
obs[OFF_ROUND + rnd.round_number] = 1.0
|
||||
obs[OFF_PHASE] = 0.0 if rnd.is_guessing_completed() else 1.0
|
||||
|
||||
for seat, guess in rnd.guesses.items():
|
||||
rel = relative_seat(seat, player)
|
||||
obs[OFF_GUESSES + 2 * rel] = 1.0
|
||||
obs[OFF_GUESSES + 2 * rel + 1] = guess / 8
|
||||
|
||||
tricks = rnd.get_stashes_winner_summary()
|
||||
for seat in range(4):
|
||||
obs[OFF_TRICKS + relative_seat(seat, player)] = tricks[seat] / 8
|
||||
|
||||
if current_stash is not None:
|
||||
for seat, card in current_stash.get_cards().items():
|
||||
rel = relative_seat(seat, player)
|
||||
obs[OFF_STASH + rel * N_CARDS + card_index(card)] = 1.0
|
||||
obs[OFF_STASH_LEADER + relative_seat(current_stash.first_player, player)] = 1.0
|
||||
|
||||
for seat, banned in deduce_voids(rnd).items():
|
||||
rel = relative_seat(seat, player)
|
||||
for color in banned:
|
||||
obs[OFF_VOIDS + rel * 4 + _COLOR_INDEX[color]] = 1.0
|
||||
|
||||
return obs
|
||||
|
||||
|
||||
def guess_mask(rnd) -> list:
|
||||
"""Maska legalnych tipov (N_GUESS_ACTIONS boolov) pre aktivneho tipujuceho.
|
||||
|
||||
Zrkadli Round.add_player_guess: tip 0..(8 - round_number); poslednemu
|
||||
(stvrtemu) tipujucemu je navyse zakazana hodnota, pri ktorej by sucet
|
||||
tipov vysiel presne na pocet kopiek v kole.
|
||||
"""
|
||||
max_guess = 8 - rnd.round_number
|
||||
mask = [g <= max_guess for g in range(N_GUESS_ACTIONS)]
|
||||
if len(rnd.guesses) == 3:
|
||||
forbidden = max_guess - sum(rnd.guesses.values())
|
||||
if 0 <= forbidden <= max_guess:
|
||||
mask[forbidden] = False
|
||||
return mask
|
||||
|
||||
|
||||
def legal_cards(hand: list, first_card) -> list:
|
||||
"""Karty z `hand`, ktore smie hrac zahrat do kopky vynasanej `first_card`.
|
||||
|
||||
Zrkadli follow-suit logiku Round.play_card: povinna farba prvej karty
|
||||
kopky; ak ju hrac nema, povinna cervena (tromf); ak nema ani tu,
|
||||
lubovolna karta. Pri vynasani (`first_card is None`) lubovolna karta.
|
||||
"""
|
||||
if first_card is None:
|
||||
return list(hand)
|
||||
same_color = [c for c in hand if c.color == first_card.color]
|
||||
if same_color:
|
||||
return same_color
|
||||
hearts = [c for c in hand if c.color == Card_colors['HEARTS']]
|
||||
return hearts if hearts else list(hand)
|
||||
|
||||
|
||||
def play_mask(rnd, player: int) -> list:
|
||||
"""Maska legalnych kariet (N_PLAY_ACTIONS boolov) pre hraca `player`."""
|
||||
stash = rnd.get_last_stash()
|
||||
first_card = stash.get_first_card() if stash is not None else None
|
||||
mask = [False] * N_PLAY_ACTIONS
|
||||
for card in legal_cards(rnd.player_cards[player], first_card):
|
||||
mask[card_index(card)] = True
|
||||
return mask
|
||||
@@ -0,0 +1,70 @@
|
||||
"""Self-play prostredie nad jednym `Round`-om (viz rl/DESIGN.md).
|
||||
|
||||
Jedno kolo = jedna epizoda. Prostredie je multi-agentne a tahove: v kazdom
|
||||
kroku je na tahu prave jeden hrac (`Decision.player`), akciu zan doda
|
||||
volajuci (zdielana siet, heuristika, ...). Odmena je sparse a terminalna --
|
||||
`Round.get_points_summary()` pre vsetkych 4 hracov naraz na konci kola.
|
||||
|
||||
Akcie: v tipovacej faze index tipu 0..8, v hracej faze index karty 0..31
|
||||
(kanonicke cislovanie z rl/encoding.py). Legalne akcie urcuje
|
||||
`Decision.mask`; nelegalna akcia prebuble ako BridzikException z enginu.
|
||||
"""
|
||||
|
||||
from random import Random
|
||||
from typing import NamedTuple
|
||||
|
||||
from bridzik import Round, ROUNDS_PER_SERIES
|
||||
from rl.encoding import encode_observation, guess_mask, index_card, play_mask
|
||||
|
||||
PHASE_GUESS = 'guess'
|
||||
PHASE_PLAY = 'play'
|
||||
|
||||
|
||||
class Decision(NamedTuple):
|
||||
"""Jeden rozhodovaci bod: kto je na tahu, v akej faze, co vidi a co smie."""
|
||||
player: int
|
||||
phase: str
|
||||
obs: list
|
||||
mask: list
|
||||
|
||||
|
||||
class RoundEnv:
|
||||
def __init__(self, rng: Random = None):
|
||||
self.rng = rng if rng is not None else Random()
|
||||
self.round = None
|
||||
|
||||
def reset(self, round_number: int = None, first_player: int = None) -> Decision:
|
||||
"""Zacne novu epizodu; nezadane parametre sa sampluju uniformne."""
|
||||
if round_number is None:
|
||||
round_number = self.rng.randrange(ROUNDS_PER_SERIES)
|
||||
if first_player is None:
|
||||
first_player = self.rng.randrange(4)
|
||||
self.round = Round(round_number, first_player, shuffler=self.rng.shuffle)
|
||||
return self._decision()
|
||||
|
||||
def step(self, action: int):
|
||||
"""Vykona akciu hraca na tahu.
|
||||
|
||||
Vracia (decision, rewards, done): pocas kola (Decision, None, False),
|
||||
na konci kola (None, [body 4 hracov], True).
|
||||
"""
|
||||
if self.round is None or self.round.is_completed():
|
||||
raise RuntimeError('Epizoda nebezi -- najprv zavolaj reset().')
|
||||
player = self.round.get_active_player()
|
||||
if not self.round.is_guessing_completed():
|
||||
self.round.add_player_guess(player, action)
|
||||
else:
|
||||
self.round.play_card(player, index_card(action))
|
||||
if self.round.is_completed():
|
||||
return None, self.round.get_points_summary(), True
|
||||
return self._decision(), None, False
|
||||
|
||||
def _decision(self) -> Decision:
|
||||
player = self.round.get_active_player()
|
||||
if not self.round.is_guessing_completed():
|
||||
return Decision(player, PHASE_GUESS,
|
||||
encode_observation(self.round, player),
|
||||
guess_mask(self.round))
|
||||
return Decision(player, PHASE_PLAY,
|
||||
encode_observation(self.round, player),
|
||||
play_mask(self.round, player))
|
||||
@@ -0,0 +1,296 @@
|
||||
import copy
|
||||
import random
|
||||
import unittest
|
||||
|
||||
from bridzik import cards, Card, Card_colors, Card_values, BridzikException, Round
|
||||
from rl.encoding import (
|
||||
COLORS, VALUES, N_CARDS, N_GUESS_ACTIONS, N_PLAY_ACTIONS,
|
||||
OFF_HAND, OFF_SEEN, OFF_ROUND, OFF_PHASE, OFF_GUESSES, OFF_TRICKS,
|
||||
OFF_STASH, OFF_STASH_LEADER, OFF_VOIDS, OBS_DIM,
|
||||
card_index, deduce_voids, index_card, relative_seat, encode_observation,
|
||||
guess_mask, play_mask,
|
||||
)
|
||||
|
||||
|
||||
class CardIndexCase(unittest.TestCase):
|
||||
def test_roundtrip_and_uniqueness(self):
|
||||
indexes = set()
|
||||
for card in cards:
|
||||
idx = card_index(card)
|
||||
self.assertIn(idx, range(N_CARDS))
|
||||
self.assertEqual(index_card(idx), card)
|
||||
indexes.add(idx)
|
||||
self.assertEqual(len(indexes), N_CARDS)
|
||||
|
||||
def test_layout(self):
|
||||
# farba = blok po 8, hodnota = pozicia v bloku
|
||||
self.assertEqual(card_index(Card(Card_colors['HEARTS'], Card_values['C7'])), 0)
|
||||
self.assertEqual(card_index(Card(Card_colors['HEARTS'], Card_values['ACE'])), 7)
|
||||
self.assertEqual(card_index(Card(COLORS[3], Card_values['ACE'])), 31)
|
||||
|
||||
|
||||
class RotationCase(unittest.TestCase):
|
||||
def test_relative_seat(self):
|
||||
for player in range(4):
|
||||
self.assertEqual(relative_seat(player, player), 0)
|
||||
# smer hry = rastuce cislo sedadla mod 4
|
||||
self.assertEqual(relative_seat((player + 1) % 4, player), 1)
|
||||
self.assertEqual(relative_seat((player + 3) % 4, player), 3)
|
||||
|
||||
def test_guesses_rotated_for_all_seats(self):
|
||||
r = Round(0, 2)
|
||||
guesses = {2: 5, 3: 0, 0: 1, 1: 1}
|
||||
for seat in [2, 3, 0, 1]:
|
||||
r.add_player_guess(seat, guesses[seat])
|
||||
for player in range(4):
|
||||
obs = encode_observation(r, player)
|
||||
for seat in range(4):
|
||||
rel = relative_seat(seat, player)
|
||||
self.assertEqual(obs[OFF_GUESSES + 2 * rel], 1.0)
|
||||
self.assertEqual(obs[OFF_GUESSES + 2 * rel + 1], guesses[seat] / 8)
|
||||
|
||||
def test_partial_guesses_flags(self):
|
||||
r = Round(3, 1)
|
||||
r.add_player_guess(1, 2)
|
||||
for player in range(4):
|
||||
obs = encode_observation(r, player)
|
||||
rel = relative_seat(1, player)
|
||||
self.assertEqual(obs[OFF_GUESSES + 2 * rel], 1.0)
|
||||
self.assertEqual(obs[OFF_GUESSES + 2 * rel + 1], 2 / 8)
|
||||
for seat in [0, 2, 3]:
|
||||
rel = relative_seat(seat, player)
|
||||
self.assertEqual(obs[OFF_GUESSES + 2 * rel], 0.0)
|
||||
self.assertEqual(obs[OFF_GUESSES + 2 * rel + 1], 0.0)
|
||||
|
||||
|
||||
class ObservationCase(unittest.TestCase):
|
||||
@staticmethod
|
||||
def _deterministic_round():
|
||||
# rovnaka konstrukcia ako v test_engine.RoundCase.test_play_card
|
||||
shuffler = lambda l: None
|
||||
c0 = [Card(Card_colors['BELLS'], Card_values['UPPER']),
|
||||
Card(Card_colors['HEARTS'], Card_values['UPPER'])]
|
||||
c1 = [Card(Card_colors['BELLS'], Card_values['C7']),
|
||||
Card(Card_colors['HEARTS'], Card_values['C10'])]
|
||||
c2 = [Card(Card_colors['BELLS'], Card_values['ACE']),
|
||||
Card(Card_colors['BELLS'], Card_values['C8'])]
|
||||
c3 = [Card(Card_colors['LEAVES'], Card_values['C7']),
|
||||
Card(Card_colors['BELLS'], Card_values['LOWER'])]
|
||||
c = ['dummy'] * 24 + c0 + c1 + c2 + c3
|
||||
r = Round(6, 1, c, shuffler)
|
||||
return r, [c0, c1, c2, c3]
|
||||
|
||||
def test_hand_multi_hot(self):
|
||||
r, hands = self._deterministic_round()
|
||||
for player in range(4):
|
||||
obs = encode_observation(r, player)
|
||||
hand_indexes = {card_index(c) for c in hands[player]}
|
||||
for i in range(N_CARDS):
|
||||
self.assertEqual(obs[OFF_HAND + i], 1.0 if i in hand_indexes else 0.0)
|
||||
|
||||
def test_round_number_and_phase(self):
|
||||
r, _ = self._deterministic_round()
|
||||
obs = encode_observation(r, 0)
|
||||
for i in range(8):
|
||||
self.assertEqual(obs[OFF_ROUND + i], 1.0 if i == 6 else 0.0)
|
||||
self.assertEqual(obs[OFF_PHASE], 1.0) # tipovacia faza
|
||||
|
||||
for seat, guess in [(1, 0), (2, 0), (3, 1), (0, 2)]:
|
||||
r.add_player_guess(seat, guess)
|
||||
obs = encode_observation(r, 0)
|
||||
self.assertEqual(obs[OFF_PHASE], 0.0) # hracia faza
|
||||
|
||||
def test_current_stash_slots_and_seen(self):
|
||||
r, hands = self._deterministic_round()
|
||||
for seat, guess in [(1, 0), (2, 0), (3, 1), (0, 2)]:
|
||||
r.add_player_guess(seat, guess)
|
||||
|
||||
# rozohrana kopka: hraju 0 a 1
|
||||
r.play_card(0, hands[0][0])
|
||||
r.play_card(1, hands[1][0])
|
||||
for player in range(4):
|
||||
obs = encode_observation(r, player)
|
||||
slot0 = relative_seat(0, player)
|
||||
slot1 = relative_seat(1, player)
|
||||
self.assertEqual(obs[OFF_STASH + slot0 * N_CARDS + card_index(hands[0][0])], 1.0)
|
||||
self.assertEqual(obs[OFF_STASH + slot1 * N_CARDS + card_index(hands[1][0])], 1.0)
|
||||
self.assertEqual(sum(obs[OFF_STASH:OFF_STASH + 4 * N_CARDS]), 2.0)
|
||||
# leader kopky je hrac 0 (najvyssi tip)
|
||||
self.assertEqual(obs[OFF_STASH_LEADER + relative_seat(0, player)], 1.0)
|
||||
# nic este nie je "videne" -- prva kopka nie je dokoncena
|
||||
self.assertEqual(sum(obs[OFF_SEEN:OFF_SEEN + N_CARDS]), 0.0)
|
||||
|
||||
# dokoncena kopka -> karty sa presunu do SEEN, sloty sa vyprazdnia
|
||||
r.play_card(2, hands[2][0])
|
||||
r.play_card(3, hands[3][1])
|
||||
obs = encode_observation(r, 0)
|
||||
played = [hands[0][0], hands[1][0], hands[2][0], hands[3][1]]
|
||||
for card in played:
|
||||
self.assertEqual(obs[OFF_SEEN + card_index(card)], 1.0)
|
||||
self.assertEqual(sum(obs[OFF_SEEN:OFF_SEEN + N_CARDS]), 4.0)
|
||||
self.assertEqual(sum(obs[OFF_STASH:OFF_STASH + 4 * N_CARDS]), 0.0)
|
||||
# novu kopku vynasa vitaz (hrac 2, BELLS ACE)
|
||||
self.assertEqual(obs[OFF_STASH_LEADER + relative_seat(2, 0)], 1.0)
|
||||
# pocty vyhranych kopiek rotovane
|
||||
for player in range(4):
|
||||
obs = encode_observation(r, player)
|
||||
self.assertEqual(obs[OFF_TRICKS + relative_seat(2, player)], 1 / 8)
|
||||
|
||||
def test_terminal_state_encodable(self):
|
||||
r = Round(7, 0)
|
||||
for seat, guess in [(0, 0), (1, 0), (2, 0), (3, 0)]:
|
||||
try:
|
||||
r.add_player_guess(seat, guess)
|
||||
except BridzikException:
|
||||
r.add_player_guess(seat, 1)
|
||||
while not r.is_completed():
|
||||
player = r.get_active_player()
|
||||
mask = play_mask(r, player)
|
||||
r.play_card(player, index_card(mask.index(True)))
|
||||
obs = encode_observation(r, 0)
|
||||
self.assertEqual(len(obs), OBS_DIM)
|
||||
self.assertEqual(sum(obs[OFF_SEEN:OFF_SEEN + N_CARDS]), 4.0)
|
||||
|
||||
|
||||
class VoidsInObservationCase(unittest.TestCase):
|
||||
def test_voids_encoded_and_rotated(self):
|
||||
# hrac 0 vynasa zelen; 2 tromfne (void zelen), 3 hodi gulu (void
|
||||
# zelen aj cerven) -- viz deduce_voids
|
||||
hand0 = [Card(Card_colors['LEAVES'], Card_values['C7']),
|
||||
Card(Card_colors['LEAVES'], Card_values['C8'])]
|
||||
hand1 = [Card(Card_colors['LEAVES'], Card_values['C9']),
|
||||
Card(Card_colors['LEAVES'], Card_values['C10'])]
|
||||
hand2 = [Card(Card_colors['HEARTS'], Card_values['C7']),
|
||||
Card(Card_colors['ACORNS'], Card_values['C7'])]
|
||||
hand3 = [Card(Card_colors['BELLS'], Card_values['C7']),
|
||||
Card(Card_colors['BELLS'], Card_values['C8'])]
|
||||
rest = [c for c in cards if c not in hand0 + hand1 + hand2 + hand3]
|
||||
deck = rest[:24] + hand0 + hand1 + hand2 + hand3
|
||||
r = Round(6, 0, deck, shuffler=lambda l: None)
|
||||
r.add_player_guess(0, 2)
|
||||
r.add_player_guess(1, 0)
|
||||
r.add_player_guess(2, 0)
|
||||
r.add_player_guess(3, 1)
|
||||
|
||||
obs = encode_observation(r, 0)
|
||||
self.assertEqual(sum(obs[OFF_VOIDS:OFF_VOIDS + 16]), 0.0)
|
||||
|
||||
for seat, card in [(0, hand0[0]), (1, hand1[0]),
|
||||
(2, hand2[0]), (3, hand3[0])]:
|
||||
r.play_card(seat, card)
|
||||
|
||||
leaves_i = COLORS.index(Card_colors['LEAVES'])
|
||||
hearts_i = COLORS.index(Card_colors['HEARTS'])
|
||||
for player in range(4):
|
||||
obs = encode_observation(r, player)
|
||||
block = lambda seat: obs[OFF_VOIDS + relative_seat(seat, player) * 4:
|
||||
OFF_VOIDS + relative_seat(seat, player) * 4 + 4]
|
||||
self.assertEqual(sum(block(0)), 0.0) # vynasajuci neprezradza nic
|
||||
self.assertEqual(sum(block(1)), 0.0) # priznal farbu
|
||||
self.assertEqual(block(2)[leaves_i], 1.0)
|
||||
self.assertEqual(sum(block(2)), 1.0)
|
||||
self.assertEqual(block(3)[leaves_i], 1.0)
|
||||
self.assertEqual(block(3)[hearts_i], 1.0)
|
||||
self.assertEqual(sum(block(3)), 2.0)
|
||||
|
||||
|
||||
class GuessMaskCase(unittest.TestCase):
|
||||
def test_range_by_round_number(self):
|
||||
for round_number in range(8):
|
||||
r = Round(round_number, 0)
|
||||
mask = guess_mask(r)
|
||||
for g in range(N_GUESS_ACTIONS):
|
||||
self.assertEqual(mask[g], g <= 8 - round_number)
|
||||
|
||||
def test_last_guesser_forbidden_value(self):
|
||||
r = Round(0, 0)
|
||||
r.add_player_guess(0, 2)
|
||||
r.add_player_guess(1, 1)
|
||||
r.add_player_guess(2, 3)
|
||||
mask = guess_mask(r)
|
||||
self.assertFalse(mask[2]) # 2+1+3+2 == 8 kopiek -> zakazane
|
||||
for g in [0, 1, 3, 4, 5, 6, 7, 8]:
|
||||
self.assertTrue(mask[g])
|
||||
|
||||
def test_forbidden_value_out_of_range(self):
|
||||
# sucet tipov > pocet kopiek -> zakazana hodnota by bola zaporna,
|
||||
# ziadne dodatocne maskovanie
|
||||
r = Round(0, 0)
|
||||
r.add_player_guess(0, 8)
|
||||
r.add_player_guess(1, 5)
|
||||
r.add_player_guess(2, 0)
|
||||
mask = guess_mask(r)
|
||||
self.assertEqual(mask, [True] * 9)
|
||||
|
||||
|
||||
class MaskEngineConsistencyCase(unittest.TestCase):
|
||||
"""Fuzz: maska presne zrkadli engine -- povolena akcia NIKDY nezlyha,
|
||||
zakazana akcia VZDY vyhodi BridzikException."""
|
||||
|
||||
def _check_guess_mask(self, rnd, player, mask):
|
||||
for g in range(N_GUESS_ACTIONS):
|
||||
if mask[g]:
|
||||
copy.deepcopy(rnd).add_player_guess(player, g)
|
||||
else:
|
||||
with self.assertRaises(BridzikException):
|
||||
rnd.add_player_guess(player, g)
|
||||
|
||||
def _check_play_mask(self, rnd, player, mask):
|
||||
self.assertIn(True, mask) # aktivny hrac ma vzdy legalny tah
|
||||
for i in range(N_PLAY_ACTIONS):
|
||||
if mask[i]:
|
||||
copy.deepcopy(rnd).play_card(player, index_card(i))
|
||||
else:
|
||||
with self.assertRaises(BridzikException):
|
||||
rnd.play_card(player, index_card(i))
|
||||
|
||||
def _check_observation(self, rnd, player):
|
||||
obs = encode_observation(rnd, player)
|
||||
self.assertEqual(len(obs), OBS_DIM)
|
||||
for v in obs:
|
||||
self.assertGreaterEqual(v, 0.0)
|
||||
self.assertLessEqual(v, 1.0)
|
||||
hand = {card_index(c) for c in rnd.player_cards[player]}
|
||||
for i in range(N_CARDS):
|
||||
self.assertEqual(obs[OFF_HAND + i], 1.0 if i in hand else 0.0)
|
||||
if i in hand: # ruka a videne karty su disjunktne
|
||||
self.assertEqual(obs[OFF_SEEN + i], 0.0)
|
||||
# zakodovany void nikdy neprotireci realnej ruke hraca
|
||||
for seat in range(4):
|
||||
rel = relative_seat(seat, player)
|
||||
held = {c.color for c in rnd.player_cards[seat]}
|
||||
for ci, color in enumerate(COLORS):
|
||||
if obs[OFF_VOIDS + rel * 4 + ci] == 1.0:
|
||||
self.assertNotIn(color, held)
|
||||
|
||||
def _fuzz_round(self, rng, round_number, first_player):
|
||||
rnd = Round(round_number, first_player)
|
||||
for _ in range(4):
|
||||
player = rnd.get_active_player()
|
||||
self._check_observation(rnd, player)
|
||||
mask = guess_mask(rnd)
|
||||
self._check_guess_mask(rnd, player, mask)
|
||||
rnd.add_player_guess(
|
||||
player, rng.choice([g for g in range(N_GUESS_ACTIONS) if mask[g]])
|
||||
)
|
||||
while not rnd.is_completed():
|
||||
player = rnd.get_active_player()
|
||||
self._check_observation(rnd, player)
|
||||
mask = play_mask(rnd, player)
|
||||
self._check_play_mask(rnd, player, mask)
|
||||
rnd.play_card(
|
||||
player, index_card(rng.choice([i for i in range(N_PLAY_ACTIONS) if mask[i]]))
|
||||
)
|
||||
# kolo dohrane do konca cisto cez masky -> bodovanie funguje
|
||||
self.assertEqual(len(rnd.get_points_summary()), 4)
|
||||
|
||||
def test_fuzz_all_round_numbers_and_seats(self):
|
||||
rng = random.Random(1337)
|
||||
for round_number in range(8):
|
||||
for first_player in range(4):
|
||||
for _ in range(3):
|
||||
self._fuzz_round(rng, round_number, first_player)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main(verbosity=2)
|
||||
Reference in New Issue
Block a user